We propose a high-speed packet forwarding architecture on general-purpose servers, in which a Network Interface Card (NIC) drives packet forwarding by direct packet transfer to other NICs via a PCIe switch. The demand for high-speed packet forwarding technology on general-purpose servers is increasing with the spread of networking concepts such as Network Function Virtualization (NFV). However, the current architecture, which processes packets by CPU, cannot achieve the similar degree of performance that hardware routers can provide because the processing capacity of the CPU and the bandwidth of the main memory constrain the performance. Our proposed method, called P2PNIC, overcomes this constraint by eliminating the CPU and the main memory from the entire packet forwarding. In the P2PNIC architecture, a NIC determines to which NIC to forward the packets and directly transfers the packets to the NIC over the PCIe. We evaluate the P2PNIC architecture by comparing it with the DPDK applications as examples of the current architecture. The evaluation shows that the P2PNIC architecture achieves 3.44 times higher throughput and up to 79% lower latency than the DPDK applications. This study offers a new approach in software-based network infrastructure for achieving comparable performance with hardware routers in the future.