diff options
| author | Linus Torvalds <torvalds@linux-foundation.org> | 2013-11-13 08:40:34 (GMT) |
|---|---|---|
| committer | Linus Torvalds <torvalds@linux-foundation.org> | 2013-11-13 08:40:34 (GMT) |
| commit | 42a2d923cc349583ebf6fdd52a7d35e1c2f7e6bd (patch) | |
| tree | 2b2b0c03b5389c1301800119333967efafd994ca /drivers/net/ethernet/intel | |
| parent | 5cbb3d216e2041700231bcfc383ee5f8b7fc8b74 (diff) | |
| parent | 75ecab1df14d90e86cebef9ec5c76befde46e65f (diff) | |
| download | linux-fsl-qoriq-42a2d923cc349583ebf6fdd52a7d35e1c2f7e6bd.tar.xz | |
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
Pull networking updates from David Miller:
1) The addition of nftables. No longer will we need protocol aware
firewall filtering modules, it can all live in userspace.
At the core of nftables is a, for lack of a better term, virtual
machine that executes byte codes to inspect packet or metadata
(arriving interface index, etc.) and make verdict decisions.
Besides support for loading packet contents and comparing them, the
interpreter supports lookups in various datastructures as
fundamental operations. For example sets are supports, and
therefore one could create a set of whitelist IP address entries
which have ACCEPT verdicts attached to them, and use the appropriate
byte codes to do such lookups.
Since the interpreted code is composed in userspace, userspace can
do things like optimize things before giving it to the kernel.
Another major improvement is the capability of atomically updating
portions of the ruleset. In the existing netfilter implementation,
one has to update the entire rule set in order to make a change and
this is very expensive.
Userspace tools exist to create nftables rules using existing
netfilter rule sets, but both kernel implementations will need to
co-exist for quite some time as we transition from the old to the
new stuff.
Kudos to Patrick McHardy, Pablo Neira Ayuso, and others who have
worked so hard on this.
2) Daniel Borkmann and Hannes Frederic Sowa made several improvements
to our pseudo-random number generator, mostly used for things like
UDP port randomization and netfitler, amongst other things.
In particular the taus88 generater is updated to taus113, and test
cases are added.
3) Support 64-bit rates in HTB and TBF schedulers, from Eric Dumazet
and Yang Yingliang.
4) Add support for new 577xx tigon3 chips to tg3 driver, from Nithin
Sujir.
5) Fix two fatal flaws in TCP dynamic right sizing, from Eric Dumazet,
Neal Cardwell, and Yuchung Cheng.
6) Allow IP_TOS and IP_TTL to be specified in sendmsg() ancillary
control message data, much like other socket option attributes.
From Francesco Fusco.
7) Allow applications to specify a cap on the rate computed
automatically by the kernel for pacing flows, via a new
SO_MAX_PACING_RATE socket option. From Eric Dumazet.
8) Make the initial autotuned send buffer sizing in TCP more closely
reflect actual needs, from Eric Dumazet.
9) Currently early socket demux only happens for TCP sockets, but we
can do it for connected UDP sockets too. Implementation from Shawn
Bohrer.
10) Refactor inet socket demux with the goal of improving hash demux
performance for listening sockets. With the main goals being able
to use RCU lookups on even request sockets, and eliminating the
listening lock contention. From Eric Dumazet.
11) The bonding layer has many demuxes in it's fast path, and an RCU
conversion was started back in 3.11, several changes here extend the
RCU usage to even more locations. From Ding Tianhong and Wang
Yufen, based upon suggestions by Nikolay Aleksandrov and Veaceslav
Falico.
12) Allow stackability of segmentation offloads to, in particular, allow
segmentation offloading over tunnels. From Eric Dumazet.
13) Significantly improve the handling of secret keys we input into the
various hash functions in the inet hashtables, TCP fast open, as
well as syncookies. From Hannes Frederic Sowa. The key fundamental
operation is "net_get_random_once()" which uses static keys.
Hannes even extended this to ipv4/ipv6 fragmentation handling and
our generic flow dissector.
14) The generic driver layer takes care now to set the driver data to
NULL on device removal, so it's no longer necessary for drivers to
explicitly set it to NULL any more. Many drivers have been cleaned
up in this way, from Jingoo Han.
15) Add a BPF based packet scheduler classifier, from Daniel Borkmann.
16) Improve CRC32 interfaces and generic SKB checksum iterators so that
SCTP's checksumming can more cleanly be handled. Also from Daniel
Borkmann.
17) Add a new PMTU discovery mode, IP_PMTUDISC_INTERFACE, which forces
using the interface MTU value. This helps avoid PMTU attacks,
particularly on DNS servers. From Hannes Frederic Sowa.
18) Use generic XPS for transmit queue steering rather than internal
(re-)implementation in virtio-net. From Jason Wang.
* git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next: (1622 commits)
random32: add test cases for taus113 implementation
random32: upgrade taus88 generator to taus113 from errata paper
random32: move rnd_state to linux/random.h
random32: add prandom_reseed_late() and call when nonblocking pool becomes initialized
random32: add periodic reseeding
random32: fix off-by-one in seeding requirement
PHY: Add RTL8201CP phy_driver to realtek
xtsonic: add missing platform_set_drvdata() in xtsonic_probe()
macmace: add missing platform_set_drvdata() in mace_probe()
ethernet/arc/arc_emac: add missing platform_set_drvdata() in arc_emac_probe()
ipv6: protect for_each_sk_fl_rcu in mem_check with rcu_read_lock_bh
vlan: Implement vlan_dev_get_egress_qos_mask as an inline.
ixgbe: add warning when max_vfs is out of range.
igb: Update link modes display in ethtool
netfilter: push reasm skb through instead of original frag skbs
ip6_output: fragment outgoing reassembled skb properly
MAINTAINERS: mv643xx_eth: take over maintainership from Lennart
net_sched: tbf: support of 64bit rates
ixgbe: deleting dfwd stations out of order can cause null ptr deref
ixgbe: fix build err, num_rx_queues is only available with CONFIG_RPS
...
Diffstat (limited to 'drivers/net/ethernet/intel')
36 files changed, 2314 insertions, 1304 deletions
diff --git a/drivers/net/ethernet/intel/e100.c b/drivers/net/ethernet/intel/e100.c index ada6e21..cbaba44 100644 --- a/drivers/net/ethernet/intel/e100.c +++ b/drivers/net/ethernet/intel/e100.c @@ -2985,7 +2985,6 @@ err_out_free_res: err_out_disable_pdev: pci_disable_device(pdev); err_out_free_dev: - pci_set_drvdata(pdev, NULL); free_netdev(netdev); return err; } @@ -3003,7 +3002,6 @@ static void e100_remove(struct pci_dev *pdev) free_netdev(netdev); pci_release_regions(pdev); pci_disable_device(pdev); - pci_set_drvdata(pdev, NULL); } } diff --git a/drivers/net/ethernet/intel/e1000/e1000.h b/drivers/net/ethernet/intel/e1000/e1000.h index 26d9cd5..58c1472 100644 --- a/drivers/net/ethernet/intel/e1000/e1000.h +++ b/drivers/net/ethernet/intel/e1000/e1000.h @@ -325,7 +325,7 @@ enum e1000_state_t { #undef pr_fmt #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt -extern struct net_device *e1000_get_hw_dev(struct e1000_hw *hw); +struct net_device *e1000_get_hw_dev(struct e1000_hw *hw); #define e_dbg(format, arg...) \ netdev_dbg(e1000_get_hw_dev(hw), format, ## arg) #define e_err(msglvl, format, arg...) \ @@ -346,20 +346,20 @@ extern struct net_device *e1000_get_hw_dev(struct e1000_hw *hw); extern char e1000_driver_name[]; extern const char e1000_driver_version[]; -extern int e1000_up(struct e1000_adapter *adapter); -extern void e1000_down(struct e1000_adapter *adapter); -extern void e1000_reinit_locked(struct e1000_adapter *adapter); -extern void e1000_reset(struct e1000_adapter *adapter); -extern int e1000_set_spd_dplx(struct e1000_adapter *adapter, u32 spd, u8 dplx); -extern int e1000_setup_all_rx_resources(struct e1000_adapter *adapter); -extern int e1000_setup_all_tx_resources(struct e1000_adapter *adapter); -extern void e1000_free_all_rx_resources(struct e1000_adapter *adapter); -extern void e1000_free_all_tx_resources(struct e1000_adapter *adapter); -extern void e1000_update_stats(struct e1000_adapter *adapter); -extern bool e1000_has_link(struct e1000_adapter *adapter); -extern void e1000_power_up_phy(struct e1000_adapter *); -extern void e1000_set_ethtool_ops(struct net_device *netdev); -extern void e1000_check_options(struct e1000_adapter *adapter); -extern char *e1000_get_hw_dev_name(struct e1000_hw *hw); +int e1000_up(struct e1000_adapter *adapter); +void e1000_down(struct e1000_adapter *adapter); +void e1000_reinit_locked(struct e1000_adapter *adapter); +void e1000_reset(struct e1000_adapter *adapter); +int e1000_set_spd_dplx(struct e1000_adapter *adapter, u32 spd, u8 dplx); +int e1000_setup_all_rx_resources(struct e1000_adapter *adapter); +int e1000_setup_all_tx_resources(struct e1000_adapter *adapter); +void e1000_free_all_rx_resources(struct e1000_adapter *adapter); +void e1000_free_all_tx_resources(struct e1000_adapter *adapter); +void e1000_update_stats(struct e1000_adapter *adapter); +bool e1000_has_link(struct e1000_adapter *adapter); +void e1000_power_up_phy(struct e1000_adapter *); +void e1000_set_ethtool_ops(struct net_device *netdev); +void e1000_check_options(struct e1000_adapter *adapter); +char *e1000_get_hw_dev_name(struct e1000_hw *hw); #endif /* _E1000_H_ */ diff --git a/drivers/net/ethernet/intel/e1000/e1000_main.c b/drivers/net/ethernet/intel/e1000/e1000_main.c index 59ad007..ad6800a 100644 --- a/drivers/net/ethernet/intel/e1000/e1000_main.c +++ b/drivers/net/ethernet/intel/e1000/e1000_main.c @@ -3917,8 +3917,7 @@ static bool e1000_clean_tx_irq(struct e1000_adapter *adapter, " next_to_watch <%x>\n" " jiffies <%lx>\n" " next_to_watch.status <%x>\n", - (unsigned long)((tx_ring - adapter->tx_ring) / - sizeof(struct e1000_tx_ring)), + (unsigned long)(tx_ring - adapter->tx_ring), readl(hw->hw_addr + tx_ring->tdh), readl(hw->hw_addr + tx_ring->tdt), tx_ring->next_to_use, diff --git a/drivers/net/ethernet/intel/e1000e/e1000.h b/drivers/net/ethernet/intel/e1000e/e1000.h index ad0edd1..0150f7f 100644 --- a/drivers/net/ethernet/intel/e1000e/e1000.h +++ b/drivers/net/ethernet/intel/e1000e/e1000.h @@ -472,26 +472,25 @@ enum latency_range { extern char e1000e_driver_name[]; extern const char e1000e_driver_version[]; -extern void e1000e_check_options(struct e1000_adapter *adapter); -extern void e1000e_set_ethtool_ops(struct net_device *netdev); - -extern int e1000e_up(struct e1000_adapter *adapter); -extern void e1000e_down(struct e1000_adapter *adapter); -extern void e1000e_reinit_locked(struct e1000_adapter *adapter); -extern void e1000e_reset(struct e1000_adapter *adapter); -extern void e1000e_power_up_phy(struct e1000_adapter *adapter); -extern int e1000e_setup_rx_resources(struct e1000_ring *ring); -extern int e1000e_setup_tx_resources(struct e1000_ring *ring); -extern void e1000e_free_rx_resources(struct e1000_ring *ring); -extern void e1000e_free_tx_resources(struct e1000_ring *ring); -extern struct rtnl_link_stats64 *e1000e_get_stats64(struct net_device *netdev, - struct rtnl_link_stats64 - *stats); -extern void e1000e_set_interrupt_capability(struct e1000_adapter *adapter); -extern void e1000e_reset_interrupt_capability(struct e1000_adapter *adapter); -extern void e1000e_get_hw_control(struct e1000_adapter *adapter); -extern void e1000e_release_hw_control(struct e1000_adapter *adapter); -extern void e1000e_write_itr(struct e1000_adapter *adapter, u32 itr); +void e1000e_check_options(struct e1000_adapter *adapter); +void e1000e_set_ethtool_ops(struct net_device *netdev); + +int e1000e_up(struct e1000_adapter *adapter); +void e1000e_down(struct e1000_adapter *adapter); +void e1000e_reinit_locked(struct e1000_adapter *adapter); +void e1000e_reset(struct e1000_adapter *adapter); +void e1000e_power_up_phy(struct e1000_adapter *adapter); +int e1000e_setup_rx_resources(struct e1000_ring *ring); +int e1000e_setup_tx_resources(struct e1000_ring *ring); +void e1000e_free_rx_resources(struct e1000_ring *ring); +void e1000e_free_tx_resources(struct e1000_ring *ring); +struct rtnl_link_stats64 *e1000e_get_stats64(struct net_device *netdev, + struct rtnl_link_stats64 *stats); +void e1000e_set_interrupt_capability(struct e1000_adapter *adapter); +void e1000e_reset_interrupt_capability(struct e1000_adapter *adapter); +void e1000e_get_hw_control(struct e1000_adapter *adapter); +void e1000e_release_hw_control(struct e1000_adapter *adapter); +void e1000e_write_itr(struct e1000_adapter *adapter, u32 itr); extern unsigned int copybreak; @@ -508,8 +507,8 @@ extern const struct e1000_info e1000_pch2_info; extern const struct e1000_info e1000_pch_lpt_info; extern const struct e1000_info e1000_es2_info; -extern void e1000e_ptp_init(struct e1000_adapter *adapter); -extern void e1000e_ptp_remove(struct e1000_adapter *adapter); +void e1000e_ptp_init(struct e1000_adapter *adapter); +void e1000e_ptp_remove(struct e1000_adapter *adapter); static inline s32 e1000_phy_hw_reset(struct e1000_hw *hw) { @@ -536,7 +535,7 @@ static inline s32 e1e_wphy_locked(struct e1000_hw *hw, u32 offset, u16 data) return hw->phy.ops.write_reg_locked(hw, offset, data); } -extern void e1000e_reload_nvm_generic(struct e1000_hw *hw); +void e1000e_reload_nvm_generic(struct e1000_hw *hw); static inline s32 e1000e_read_mac_addr(struct e1000_hw *hw) { diff --git a/drivers/net/ethernet/intel/i40e/i40e.h b/drivers/net/ethernet/intel/i40e/i40e.h index b5252eb..1ca9834 100644 --- a/drivers/net/ethernet/intel/i40e/i40e.h +++ b/drivers/net/ethernet/intel/i40e/i40e.h @@ -46,7 +46,6 @@ #include <linux/sctp.h> #include <linux/pkt_sched.h> #include <linux/ipv6.h> -#include <linux/version.h> #include <net/checksum.h> #include <net/ip6_checksum.h> #include <linux/ethtool.h> @@ -347,9 +346,9 @@ struct i40e_vsi { u32 rx_buf_failed; u32 rx_page_failed; - /* These are arrays of rings, allocated at run-time */ - struct i40e_ring *rx_rings; - struct i40e_ring *tx_rings; + /* These are containers of ring pointers, allocated at run-time */ + struct i40e_ring **rx_rings; + struct i40e_ring **tx_rings; u16 work_limit; /* high bit set means dynamic, use accessor routines to read/write. @@ -366,7 +365,7 @@ struct i40e_vsi { u8 dtype; /* List of q_vectors allocated to this VSI */ - struct i40e_q_vector *q_vectors; + struct i40e_q_vector **q_vectors; int num_q_vectors; int base_vector; @@ -422,8 +421,9 @@ struct i40e_q_vector { u8 num_ringpairs; /* total number of ring pairs in vector */ - char name[IFNAMSIZ + 9]; cpumask_t affinity_mask; + struct rcu_head rcu; /* to avoid race with update stats on free */ + char name[IFNAMSIZ + 9]; } ____cacheline_internodealigned_in_smp; /* lan device */ @@ -544,6 +544,7 @@ static inline void i40e_dbg_init(void) {} static inline void i40e_dbg_exit(void) {} #endif /* CONFIG_DEBUG_FS*/ void i40e_irq_dynamic_enable(struct i40e_vsi *vsi, int vector); +void i40e_irq_dynamic_enable_icr0(struct i40e_pf *pf); int i40e_ioctl(struct net_device *netdev, struct ifreq *ifr, int cmd); void i40e_vlan_stripping_disable(struct i40e_vsi *vsi); int i40e_vsi_add_vlan(struct i40e_vsi *vsi, s16 vid); diff --git a/drivers/net/ethernet/intel/i40e/i40e_debugfs.c b/drivers/net/ethernet/intel/i40e/i40e_debugfs.c index 8dbd91f..ef4cb1c 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_debugfs.c +++ b/drivers/net/ethernet/intel/i40e/i40e_debugfs.c @@ -151,9 +151,7 @@ static ssize_t i40e_dbg_dump_write(struct file *filp, size_t count, loff_t *ppos) { struct i40e_pf *pf = filp->private_data; - char dump_request_buf[16]; bool seid_found = false; - int bytes_not_copied; long seid = -1; int buflen = 0; int i, ret; @@ -163,21 +161,12 @@ static ssize_t i40e_dbg_dump_write(struct file *filp, /* don't allow partial writes */ if (*ppos != 0) return 0; - if (count >= sizeof(dump_request_buf)) - return -ENOSPC; - - bytes_not_copied = copy_from_user(dump_request_buf, buffer, count); - if (bytes_not_copied < 0) - return bytes_not_copied; - if (bytes_not_copied > 0) - count -= bytes_not_copied; - dump_request_buf[count] = '\0'; /* decode the SEID given to be dumped */ - ret = kstrtol(dump_request_buf, 0, &seid); - if (ret < 0) { - dev_info(&pf->pdev->dev, "bad seid value '%s'\n", - dump_request_buf); + ret = kstrtol_from_user(buffer, count, 0, &seid); + + if (ret) { + dev_info(&pf->pdev->dev, "bad seid value\n"); } else if (seid == 0) { seid_found = true; @@ -245,26 +234,33 @@ static ssize_t i40e_dbg_dump_write(struct file *filp, memcpy(p, vsi, len); p += len; - len = (sizeof(struct i40e_q_vector) - * vsi->num_q_vectors); - memcpy(p, vsi->q_vectors, len); - p += len; - - len = (sizeof(struct i40e_ring) * vsi->num_queue_pairs); - memcpy(p, vsi->tx_rings, len); - p += len; - memcpy(p, vsi->rx_rings, len); - p += len; + if (vsi->num_q_vectors) { + len = (sizeof(struct i40e_q_vector) + * vsi->num_q_vectors); + memcpy(p, vsi->q_vectors, len); + p += len; + } - for (i = 0; i < vsi->num_queue_pairs; i++) { - len = sizeof(struct i40e_tx_buffer); - memcpy(p, vsi->tx_rings[i].tx_bi, len); + if (vsi->num_queue_pairs) { + len = (sizeof(struct i40e_ring) * + vsi->num_queue_pairs); + memcpy(p, vsi->tx_rings, len); + p += len; + memcpy(p, vsi->rx_rings, len); p += len; } - for (i = 0; i < vsi->num_queue_pairs; i++) { + + if (vsi->tx_rings[0]) { + len = sizeof(struct i40e_tx_buffer); + for (i = 0; i < vsi->num_queue_pairs; i++) { + memcpy(p, vsi->tx_rings[i]->tx_bi, len); + p += len; + } len = sizeof(struct i40e_rx_buffer); - memcpy(p, vsi->rx_rings[i].rx_bi, len); - p += len; + for (i = 0; i < vsi->num_queue_pairs; i++) { + memcpy(p, vsi->rx_rings[i]->rx_bi, len); + p += len; + } } /* macvlan filter list */ @@ -484,100 +480,104 @@ static void i40e_dbg_dump_vsi_seid(struct i40e_pf *pf, int seid) " tx_restart = %d, tx_busy = %d, rx_buf_failed = %d, rx_page_failed = %d\n", vsi->tx_restart, vsi->tx_busy, vsi->rx_buf_failed, vsi->rx_page_failed); - if (vsi->rx_rings) { - for (i = 0; i < vsi->num_queue_pairs; i++) { - dev_info(&pf->pdev->dev, - " rx_rings[%i]: desc = %p\n", - i, vsi->rx_rings[i].desc); - dev_info(&pf->pdev->dev, - " rx_rings[%i]: dev = %p, netdev = %p, rx_bi = %p\n", - i, vsi->rx_rings[i].dev, - vsi->rx_rings[i].netdev, - vsi->rx_rings[i].rx_bi); - dev_info(&pf->pdev->dev, - " rx_rings[%i]: state = %li, queue_index = %d, reg_idx = %d\n", - i, vsi->rx_rings[i].state, - vsi->rx_rings[i].queue_index, - vsi->rx_rings[i].reg_idx); - dev_info(&pf->pdev->dev, - " rx_rings[%i]: rx_hdr_len = %d, rx_buf_len = %d, dtype = %d\n", - i, vsi->rx_rings[i].rx_hdr_len, - vsi->rx_rings[i].rx_buf_len, - vsi->rx_rings[i].dtype); - dev_info(&pf->pdev->dev, - " rx_rings[%i]: hsplit = %d, next_to_use = %d, next_to_clean = %d, ring_active = %i\n", - i, vsi->rx_rings[i].hsplit, - vsi->rx_rings[i].next_to_use, - vsi->rx_rings[i].next_to_clean, - vsi->rx_rings[i].ring_active); - dev_info(&pf->pdev->dev, - " rx_rings[%i]: rx_stats: packets = %lld, bytes = %lld, non_eop_descs = %lld\n", - i, vsi->rx_rings[i].rx_stats.packets, - vsi->rx_rings[i].rx_stats.bytes, - vsi->rx_rings[i].rx_stats.non_eop_descs); - dev_info(&pf->pdev->dev, - " rx_rings[%i]: rx_stats: alloc_rx_page_failed = %lld, alloc_rx_buff_failed = %lld\n", - i, - vsi->rx_rings[i].rx_stats.alloc_rx_page_failed, - vsi->rx_rings[i].rx_stats.alloc_rx_buff_failed); - dev_info(&pf->pdev->dev, - " rx_rings[%i]: size = %i, dma = 0x%08lx\n", - i, vsi->rx_rings[i].size, - (long unsigned int)vsi->rx_rings[i].dma); - dev_info(&pf->pdev->dev, - " rx_rings[%i]: vsi = %p, q_vector = %p\n", - i, vsi->rx_rings[i].vsi, - vsi->rx_rings[i].q_vector); - } + rcu_read_lock(); + for (i = 0; i < vsi->num_queue_pairs; i++) { + struct i40e_ring *rx_ring = ACCESS_ONCE(vsi->rx_rings[i]); + if (!rx_ring) + continue; + + dev_info(&pf->pdev->dev, + " rx_rings[%i]: desc = %p\n", + i, rx_ring->desc); + dev_info(&pf->pdev->dev, + " rx_rings[%i]: dev = %p, netdev = %p, rx_bi = %p\n", + i, rx_ring->dev, + rx_ring->netdev, + rx_ring->rx_bi); + dev_info(&pf->pdev->dev, + " rx_rings[%i]: state = %li, queue_index = %d, reg_idx = %d\n", + i, rx_ring->state, + rx_ring->queue_index, + rx_ring->reg_idx); + dev_info(&pf->pdev->dev, + " rx_rings[%i]: rx_hdr_len = %d, rx_buf_len = %d, dtype = %d\n", + i, rx_ring->rx_hdr_len, + rx_ring->rx_buf_len, + rx_ring->dtype); + dev_info(&pf->pdev->dev, + " rx_rings[%i]: hsplit = %d, next_to_use = %d, next_to_clean = %d, ring_active = %i\n", + i, rx_ring->hsplit, + rx_ring->next_to_use, + rx_ring->next_to_clean, + rx_ring->ring_active); + dev_info(&pf->pdev->dev, + " rx_rings[%i]: rx_stats: packets = %lld, bytes = %lld, non_eop_descs = %lld\n", + i, rx_ring->stats.packets, + rx_ring->stats.bytes, + rx_ring->rx_stats.non_eop_descs); + dev_info(&pf->pdev->dev, + " rx_rings[%i]: rx_stats: alloc_rx_page_failed = %lld, alloc_rx_buff_failed = %lld\n", + i, + rx_ring->rx_stats.alloc_rx_page_failed, + rx_ring->rx_stats.alloc_rx_buff_failed); + dev_info(&pf->pdev->dev, + " rx_rings[%i]: size = %i, dma = 0x%08lx\n", + i, rx_ring->size, + (long unsigned int)rx_ring->dma); + dev_info(&pf->pdev->dev, + " rx_rings[%i]: vsi = %p, q_vector = %p\n", + i, rx_ring->vsi, + rx_ring->q_vector); } - if (vsi->tx_rings) { - for (i = 0; i < vsi->num_queue_pairs; i++) { - dev_info(&pf->pdev->dev, - " tx_rings[%i]: desc = %p\n", - i, vsi->tx_rings[i].desc); - dev_info(&pf->pdev->dev, - " tx_rings[%i]: dev = %p, netdev = %p, tx_bi = %p\n", - i, vsi->tx_rings[i].dev, - vsi->tx_rings[i].netdev, - vsi->tx_rings[i].tx_bi); - dev_info(&pf->pdev->dev, - " tx_rings[%i]: state = %li, queue_index = %d, reg_idx = %d\n", - i, vsi->tx_rings[i].state, - vsi->tx_rings[i].queue_index, - vsi->tx_rings[i].reg_idx); - dev_info(&pf->pdev->dev, - " tx_rings[%i]: dtype = %d\n", - i, vsi->tx_rings[i].dtype); - dev_info(&pf->pdev->dev, - " tx_rings[%i]: hsplit = %d, next_to_use = %d, next_to_clean = %d, ring_active = %i\n", - i, vsi->tx_rings[i].hsplit, - vsi->tx_rings[i].next_to_use, - vsi->tx_rings[i].next_to_clean, - vsi->tx_rings[i].ring_active); - dev_info(&pf->pdev->dev, - " tx_rings[%i]: tx_stats: packets = %lld, bytes = %lld, restart_queue = %lld\n", - i, vsi->tx_rings[i].tx_stats.packets, - vsi->tx_rings[i].tx_stats.bytes, - vsi->tx_rings[i].tx_stats.restart_queue); - dev_info(&pf->pdev->dev, - " tx_rings[%i]: tx_stats: tx_busy = %lld, completed = %lld, tx_done_old = %lld\n", - i, - vsi->tx_rings[i].tx_stats.tx_busy, - vsi->tx_rings[i].tx_stats.completed, - vsi->tx_rings[i].tx_stats.tx_done_old); - dev_info(&pf->pdev->dev, - " tx_rings[%i]: size = %i, dma = 0x%08lx\n", - i, vsi->tx_rings[i].size, - (long unsigned int)vsi->tx_rings[i].dma); - dev_info(&pf->pdev->dev, - " tx_rings[%i]: vsi = %p, q_vector = %p\n", - i, vsi->tx_rings[i].vsi, - vsi->tx_rings[i].q_vector); - dev_info(&pf->pdev->dev, - " tx_rings[%i]: DCB tc = %d\n", - i, vsi->tx_rings[i].dcb_tc); - } + for (i = 0; i < vsi->num_queue_pairs; i++) { + struct i40e_ring *tx_ring = ACCESS_ONCE(vsi->tx_rings[i]); + if (!tx_ring) + continue; + dev_info(&pf->pdev->dev, + " tx_rings[%i]: desc = %p\n", + i, tx_ring->desc); + dev_info(&pf->pdev->dev, + " tx_rings[%i]: dev = %p, netdev = %p, tx_bi = %p\n", + i, tx_ring->dev, + tx_ring->netdev, + tx_ring->tx_bi); + dev_info(&pf->pdev->dev, + " tx_rings[%i]: state = %li, queue_index = %d, reg_idx = %d\n", + i, tx_ring->state, + tx_ring->queue_index, + tx_ring->reg_idx); + dev_info(&pf->pdev->dev, + " tx_rings[%i]: dtype = %d\n", + i, tx_ring->dtype); + dev_info(&pf->pdev->dev, + " tx_rings[%i]: hsplit = %d, next_to_use = %d, next_to_clean = %d, ring_active = %i\n", + i, tx_ring->hsplit, + tx_ring->next_to_use, + tx_ring->next_to_clean, + tx_ring->ring_active); + dev_info(&pf->pdev->dev, + " tx_rings[%i]: tx_stats: packets = %lld, bytes = %lld, restart_queue = %lld\n", + i, tx_ring->stats.packets, + tx_ring->stats.bytes, + tx_ring->tx_stats.restart_queue); + dev_info(&pf->pdev->dev, + " tx_rings[%i]: tx_stats: tx_busy = %lld, tx_done_old = %lld\n", + i, + tx_ring->tx_stats.tx_busy, + tx_ring->tx_stats.tx_done_old); + dev_info(&pf->pdev->dev, + " tx_rings[%i]: size = %i, dma = 0x%08lx\n", + i, tx_ring->size, + (long unsigned int)tx_ring->dma); + dev_info(&pf->pdev->dev, + " tx_rings[%i]: vsi = %p, q_vector = %p\n", + i, tx_ring->vsi, + tx_ring->q_vector); + dev_info(&pf->pdev->dev, + " tx_rings[%i]: DCB tc = %d\n", + i, tx_ring->dcb_tc); } + rcu_read_unlock(); dev_info(&pf->pdev->dev, " work_limit = %d, rx_itr_setting = %d (%s), tx_itr_setting = %d (%s)\n", vsi->work_limit, vsi->rx_itr_setting, @@ -587,15 +587,6 @@ static void i40e_dbg_dump_vsi_seid(struct i40e_pf *pf, int seid) dev_info(&pf->pdev->dev, " max_frame = %d, rx_hdr_len = %d, rx_buf_len = %d dtype = %d\n", vsi->max_frame, vsi->rx_hdr_len, vsi->rx_buf_len, vsi->dtype); - if (vsi->q_vectors) { - for (i = 0; i < vsi->num_q_vectors; i++) { - dev_info(&pf->pdev->dev, - " q_vectors[%i]: base index = %ld\n", - i, ((long int)*vsi->q_vectors[i].rx.ring- - (long int)*vsi->q_vectors[0].rx.ring)/ - sizeof(struct i40e_ring)); - } - } dev_info(&pf->pdev->dev, " num_q_vectors = %i, base_vector = %i\n", vsi->num_q_vectors, vsi->base_vector); @@ -792,9 +783,9 @@ static void i40e_dbg_dump_desc(int cnt, int vsi_seid, int ring_id, int desc_n, return; } if (is_rx_ring) - ring = vsi->rx_rings[ring_id]; + ring = *vsi->rx_rings[ring_id]; else - ring = vsi->tx_rings[ring_id]; + ring = *vsi->tx_rings[ring_id]; if (cnt == 2) { dev_info(&pf->pdev->dev, "vsi = %02i %s ring = %02i\n", vsi_seid, is_rx_ring ? "rx" : "tx", ring_id); @@ -1028,11 +1019,11 @@ static ssize_t i40e_dbg_command_write(struct file *filp, size_t count, loff_t *ppos) { struct i40e_pf *pf = filp->private_data; + char *cmd_buf, *cmd_buf_tmp; int bytes_not_copied; struct i40e_vsi *vsi; u8 *print_buf_start; u8 *print_buf; - char *cmd_buf; int vsi_seid; int veb_seid; int cnt; @@ -1051,6 +1042,12 @@ static ssize_t i40e_dbg_command_write(struct file *filp, count -= bytes_not_copied; cmd_buf[count] = '\0'; + cmd_buf_tmp = strchr(cmd_buf, '\n'); + if (cmd_buf_tmp) { + *cmd_buf_tmp = '\0'; + count = cmd_buf_tmp - cmd_buf + 1; + } + print_buf_start = kzalloc(I40E_MAX_DEBUG_OUT_BUFFER, GFP_KERNEL); if (!print_buf_start) goto command_write_done; @@ -1157,9 +1154,9 @@ static ssize_t i40e_dbg_command_write(struct file *filp, i40e_veb_release(pf->veb[i]); } else if (strncmp(cmd_buf, "add macaddr", 11) == 0) { - u8 ma[6]; - int vlan = 0; struct i40e_mac_filter *f; + int vlan = 0; + u8 ma[6]; int ret; cnt = sscanf(&cmd_buf[11], @@ -1195,8 +1192,8 @@ static ssize_t i40e_dbg_command_write(struct file *filp, ma, vlan, vsi_seid, f, ret); } else if (strncmp(cmd_buf, "del macaddr", 11) == 0) { - u8 ma[6]; int vlan = 0; + u8 ma[6]; int ret; cnt = sscanf(&cmd_buf[11], @@ -1232,9 +1229,9 @@ static ssize_t i40e_dbg_command_write(struct file *filp, ma, vlan, vsi_seid, ret); } else if (strncmp(cmd_buf, "add pvid", 8) == 0) { - int v; - u16 vid; i40e_status ret; + u16 vid; + int v; cnt = sscanf(&cmd_buf[8], "%i %u", &vsi_seid, &v); if (cnt != 2) { @@ -1545,10 +1542,10 @@ static ssize_t i40e_dbg_command_write(struct file *filp, } else if ((strncmp(cmd_buf, "add fd_filter", 13) == 0) || (strncmp(cmd_buf, "rem fd_filter", 13) == 0)) { struct i40e_fdir_data fd_data; - int ret; u16 packet_len, i, j = 0; char *asc_packet; bool add = false; + int ret; asc_packet = kzalloc(I40E_FDIR_MAX_RAW_PACKET_LOOKUP, GFP_KERNEL); @@ -1636,9 +1633,9 @@ static ssize_t i40e_dbg_command_write(struct file *filp, } } else if (strncmp(&cmd_buf[5], "get local", 9) == 0) { + u16 llen, rlen; int ret, i; u8 *buff; - u16 llen, rlen; buff = kzalloc(I40E_LLDPDU_SIZE, GFP_KERNEL); if (!buff) goto command_write_done; @@ -1669,9 +1666,9 @@ static ssize_t i40e_dbg_command_write(struct file *filp, kfree(buff); buff = NULL; } else if (strncmp(&cmd_buf[5], "get remote", 10) == 0) { + u16 llen, rlen; int ret, i; u8 *buff; - u16 llen, rlen; buff = kzalloc(I40E_LLDPDU_SIZE, GFP_KERNEL); if (!buff) goto command_write_done; @@ -1747,11 +1744,13 @@ static ssize_t i40e_dbg_command_write(struct file *filp, goto command_write_done; } - /* Read at least 512 words */ - if (buffer_len == 0) - buffer_len = 512; + /* set the max length */ + buffer_len = min_t(u16, buffer_len, I40E_MAX_AQ_BUF_SIZE/2); bytes = 2 * buffer_len; + + /* read at least 1k bytes, no more than 4kB */ + bytes = clamp(bytes, (u16)1024, (u16)I40E_MAX_AQ_BUF_SIZE); buff = kzalloc(bytes, GFP_KERNEL); if (!buff) goto command_write_done; @@ -1903,6 +1902,7 @@ static ssize_t i40e_dbg_netdev_ops_write(struct file *filp, struct i40e_pf *pf = filp->private_data; int bytes_not_copied; struct i40e_vsi *vsi; + char *buf_tmp; int vsi_seid; int i, cnt; @@ -1921,6 +1921,12 @@ static ssize_t i40e_dbg_netdev_ops_write(struct file *filp, count -= bytes_not_copied; i40e_dbg_netdev_ops_buf[count] = '\0'; + buf_tmp = strchr(i40e_dbg_netdev_ops_buf, '\n'); + if (buf_tmp) { + *buf_tmp = '\0'; + count = buf_tmp - i40e_dbg_netdev_ops_buf + 1; + } + if (strncmp(i40e_dbg_netdev_ops_buf, "tx_timeout", 10) == 0) { cnt = sscanf(&i40e_dbg_netdev_ops_buf[11], "%i", &vsi_seid); if (cnt != 1) { @@ -1996,7 +2002,7 @@ static ssize_t i40e_dbg_netdev_ops_write(struct file *filp, goto netdev_ops_write_done; } for (i = 0; i < vsi->num_q_vectors; i++) - napi_schedule(&vsi->q_vectors[i].napi); + napi_schedule(&vsi->q_vectors[i]->napi); dev_info(&pf->pdev->dev, "napi called\n"); } else { dev_info(&pf->pdev->dev, "unknown command '%s'\n", @@ -2024,21 +2030,35 @@ static const struct file_operations i40e_dbg_netdev_ops_fops = { **/ void i40e_dbg_pf_init(struct i40e_pf *pf) { - struct dentry *pfile __attribute__((unused)); + struct dentry *pfile; const char *name = pci_name(pf->pdev); + const struct device *dev = &pf->pdev->dev; pf->i40e_dbg_pf = debugfs_create_dir(name, i40e_dbg_root); - if (pf->i40e_dbg_pf) { - pfile = debugfs_create_file("command", 0600, pf->i40e_dbg_pf, - pf, &i40e_dbg_command_fops); - pfile = debugfs_create_file("dump", 0600, pf->i40e_dbg_pf, pf, - &i40e_dbg_dump_fops); - pfile = debugfs_create_file("netdev_ops", 0600, pf->i40e_dbg_pf, - pf, &i40e_dbg_netdev_ops_fops); - } else { - dev_info(&pf->pdev->dev, - "debugfs entry for %s failed\n", name); - } + if (!pf->i40e_dbg_pf) + return; + + pfile = debugfs_create_file("command", 0600, pf->i40e_dbg_pf, pf, + &i40e_dbg_command_fops); + if (!pfile) + goto create_failed; + + pfile = debugfs_create_file("dump", 0600, pf->i40e_dbg_pf, pf, + &i40e_dbg_dump_fops); + if (!pfile) + goto create_failed; + + pfile = debugfs_create_file("netdev_ops", 0600, pf->i40e_dbg_pf, pf, + &i40e_dbg_netdev_ops_fops); + if (!pfile) + goto create_failed; + + return; + +create_failed: + dev_info(dev, "debugfs dir/file for %s failed\n", name); + debugfs_remove_recursive(pf->i40e_dbg_pf); + return; } /** diff --git a/drivers/net/ethernet/intel/i40e/i40e_ethtool.c b/drivers/net/ethernet/intel/i40e/i40e_ethtool.c index 9a76b8c..1b86138 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_ethtool.c +++ b/drivers/net/ethernet/intel/i40e/i40e_ethtool.c @@ -399,8 +399,8 @@ static void i40e_get_ringparam(struct net_device *netdev, ring->tx_max_pending = I40E_MAX_NUM_DESCRIPTORS; ring->rx_mini_max_pending = 0; ring->rx_jumbo_max_pending = 0; - ring->rx_pending = vsi->rx_rings[0].count; - ring->tx_pending = vsi->tx_rings[0].count; + ring->rx_pending = vsi->rx_rings[0]->count; + ring->tx_pending = vsi->tx_rings[0]->count; ring->rx_mini_pending = 0; ring->rx_jumbo_pending = 0; } @@ -429,8 +429,8 @@ static int i40e_set_ringparam(struct net_device *netdev, new_rx_count = ALIGN(new_rx_count, I40E_REQ_DESCRIPTOR_MULTIPLE); /* if nothing to do return success */ - if ((new_tx_count == vsi->tx_rings[0].count) && - (new_rx_count == vsi->rx_rings[0].count)) + if ((new_tx_count == vsi->tx_rings[0]->count) && + (new_rx_count == vsi->rx_rings[0]->count)) return 0; while (test_and_set_bit(__I40E_CONFIG_BUSY, &pf->state)) @@ -439,8 +439,8 @@ static int i40e_set_ringparam(struct net_device *netdev, if (!netif_running(vsi->netdev)) { /* simple case - set for the next time the netdev is started */ for (i = 0; i < vsi->num_queue_pairs; i++) { - vsi->tx_rings[i].count = new_tx_count; - vsi->rx_rings[i].count = new_rx_count; + vsi->tx_rings[i]->count = new_tx_count; + vsi->rx_rings[i]->count = new_rx_count; } goto done; } @@ -451,10 +451,10 @@ static int i40e_set_ringparam(struct net_device *netdev, */ /* alloc updated Tx resources */ - if (new_tx_count != vsi->tx_rings[0].count) { + if (new_tx_count != vsi->tx_rings[0]->count) { netdev_info(netdev, "Changing Tx descriptor count from %d to %d.\n", - vsi->tx_rings[0].count, new_tx_count); + vsi->tx_rings[0]->count, new_tx_count); tx_rings = kcalloc(vsi->alloc_queue_pairs, sizeof(struct i40e_ring), GFP_KERNEL); if (!tx_rings) { @@ -464,7 +464,7 @@ static int i40e_set_ringparam(struct net_device *netdev, for (i = 0; i < vsi->num_queue_pairs; i++) { /* clone ring and setup updated count */ - tx_rings[i] = vsi->tx_rings[i]; + tx_rings[i] = *vsi->tx_rings[i]; tx_rings[i].count = new_tx_count; err = i40e_setup_tx_descriptors(&tx_rings[i]); if (err) { @@ -481,10 +481,10 @@ static int i40e_set_ringparam(struct net_device *netdev, } /* alloc updated Rx resources */ - if (new_rx_count != vsi->rx_rings[0].count) { + if (new_rx_count != vsi->rx_rings[0]->count) { netdev_info(netdev, "Changing Rx descriptor count from %d to %d\n", - vsi->rx_rings[0].count, new_rx_count); + vsi->rx_rings[0]->count, new_rx_count); rx_rings = kcalloc(vsi->alloc_queue_pairs, sizeof(struct i40e_ring), GFP_KERNEL); if (!rx_rings) { @@ -494,7 +494,7 @@ static int i40e_set_ringparam(struct net_device *netdev, for (i = 0; i < vsi->num_queue_pairs; i++) { /* clone ring and setup updated count */ - rx_rings[i] = vsi->rx_rings[i]; + rx_rings[i] = *vsi->rx_rings[i]; rx_rings[i].count = new_rx_count; err = i40e_setup_rx_descriptors(&rx_rings[i]); if (err) { @@ -517,8 +517,8 @@ static int i40e_set_ringparam(struct net_device *netdev, if (tx_rings) { for (i = 0; i < vsi->num_queue_pairs; i++) { - i40e_free_tx_resources(&vsi->tx_rings[i]); - vsi->tx_rings[i] = tx_rings[i]; + i40e_free_tx_resources(vsi->tx_rings[i]); + *vsi->tx_rings[i] = tx_rings[i]; } kfree(tx_rings); tx_rings = NULL; @@ -526,8 +526,8 @@ static int i40e_set_ringparam(struct net_device *netdev, if (rx_rings) { for (i = 0; i < vsi->num_queue_pairs; i++) { - i40e_free_rx_resources(&vsi->rx_rings[i]); - vsi->rx_rings[i] = rx_rings[i]; + i40e_free_rx_resources(vsi->rx_rings[i]); + *vsi->rx_rings[i] = rx_rings[i]; } kfree(rx_rings); rx_rings = NULL; @@ -579,6 +579,7 @@ static void i40e_get_ethtool_stats(struct net_device *netdev, char *p; int j; struct rtnl_link_stats64 *net_stats = i40e_get_vsi_stats_struct(vsi); + unsigned int start; i40e_update_stats(vsi); @@ -587,14 +588,30 @@ static void i40e_get_ethtool_stats(struct net_device *netdev, data[i++] = (i40e_gstrings_net_stats[j].sizeof_stat == sizeof(u64)) ? *(u64 *)p : *(u32 *)p; } - for (j = 0; j < vsi->num_queue_pairs; j++) { - data[i++] = vsi->tx_rings[j].tx_stats.packets; - data[i++] = vsi->tx_rings[j].tx_stats.bytes; - } - for (j = 0; j < vsi->num_queue_pairs; j++) { - data[i++] = vsi->rx_rings[j].rx_stats.packets; - data[i++] = vsi->rx_rings[j].rx_stats.bytes; + rcu_read_lock(); + for (j = 0; j < vsi->num_queue_pairs; j++, i += 4) { + struct i40e_ring *tx_ring = ACCESS_ONCE(vsi->tx_rings[j]); + struct i40e_ring *rx_ring; + + if (!tx_ring) + continue; + + /* process Tx ring statistics */ + do { + start = u64_stats_fetch_begin_bh(&tx_ring->syncp); + data[i] = tx_ring->stats.packets; + data[i + 1] = tx_ring->stats.bytes; + } while (u64_stats_fetch_retry_bh(&tx_ring->syncp, start)); + + /* Rx ring is the 2nd half of the queue pair */ + rx_ring = &tx_ring[1]; + do { + start = u64_stats_fetch_begin_bh(&rx_ring->syncp); + data[i + 2] = rx_ring->stats.packets; + data[i + 3] = rx_ring->stats.bytes; + } while (u64_stats_fetch_retry_bh(&rx_ring->syncp, start)); } + rcu_read_unlock(); if (vsi == pf->vsi[pf->lan_vsi]) { for (j = 0; j < I40E_GLOBAL_STATS_LEN; j++) { p = (char *)pf + i40e_gstrings_stats[j].stat_offset; @@ -641,8 +658,6 @@ static void i40e_get_strings(struct net_device *netdev, u32 stringset, p += ETH_GSTRING_LEN; snprintf(p, ETH_GSTRING_LEN, "tx-%u.tx_bytes", i); p += ETH_GSTRING_LEN; - } - for (i = 0; i < vsi->num_queue_pairs; i++) { snprintf(p, ETH_GSTRING_LEN, "rx-%u.rx_packets", i); p += ETH_GSTRING_LEN; snprintf(p, ETH_GSTRING_LEN, "rx-%u.rx_bytes", i); @@ -910,8 +925,8 @@ static int i40e_set_coalesce(struct net_device *netdev, } vector = vsi->base_vector; - q_vector = vsi->q_vectors; - for (i = 0; i < vsi->num_q_vectors; i++, vector++, q_vector++) { + for (i = 0; i < vsi->num_q_vectors; i++, vector++) { + q_vector = vsi->q_vectors[i]; q_vector->rx.itr = ITR_TO_REG(vsi->rx_itr_setting); wr32(hw, I40E_PFINT_ITRN(0, vector - 1), q_vector->rx.itr); q_vector->tx.itr = ITR_TO_REG(vsi->tx_itr_setting); diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c index 221aa47..be15938 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_main.c +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c @@ -36,7 +36,7 @@ static const char i40e_driver_string[] = #define DRV_VERSION_MAJOR 0 #define DRV_VERSION_MINOR 3 -#define DRV_VERSION_BUILD 9 +#define DRV_VERSION_BUILD 11 #define DRV_VERSION __stringify(DRV_VERSION_MAJOR) "." \ __stringify(DRV_VERSION_MINOR) "." \ __stringify(DRV_VERSION_BUILD) DRV_KERN @@ -347,14 +347,53 @@ struct rtnl_link_stats64 *i40e_get_vsi_stats_struct(struct i40e_vsi *vsi) **/ static struct rtnl_link_stats64 *i40e_get_netdev_stats_struct( struct net_device *netdev, - struct rtnl_link_stats64 *storage) + struct rtnl_link_stats64 *stats) { struct i40e_netdev_priv *np = netdev_priv(netdev); struct i40e_vsi *vsi = np->vsi; + struct rtnl_link_stats64 *vsi_stats = i40e_get_vsi_stats_struct(vsi); + int i; + + rcu_read_lock(); + for (i = 0; i < vsi->num_queue_pairs; i++) { + struct i40e_ring *tx_ring, *rx_ring; + u64 bytes, packets; + unsigned int start; + + tx_ring = ACCESS_ONCE(vsi->tx_rings[i]); + if (!tx_ring) + continue; - *storage = *i40e_get_vsi_stats_struct(vsi); + do { + start = u64_stats_fetch_begin_bh(&tx_ring->syncp); + packets = tx_ring->stats.packets; + bytes = tx_ring->stats.bytes; + } while (u64_stats_fetch_retry_bh(&tx_ring->syncp, start)); + + stats->tx_packets += packets; + stats->tx_bytes += bytes; + rx_ring = &tx_ring[1]; - return storage; + do { + start = u64_stats_fetch_begin_bh(&rx_ring->syncp); + packets = rx_ring->stats.packets; + bytes = rx_ring->stats.bytes; + } while (u64_stats_fetch_retry_bh(&rx_ring->syncp, start)); + + stats->rx_packets += packets; + stats->rx_bytes += bytes; + } + rcu_read_unlock(); + + /* following stats updated by ixgbe_watchdog_task() */ + stats->multicast = vsi_stats->multicast; + stats->tx_errors = vsi_stats->tx_errors; + stats->tx_dropped = vsi_stats->tx_dropped; + stats->rx_errors = vsi_stats->rx_errors; + stats->rx_crc_errors = vsi_stats->rx_crc_errors; + stats->rx_length_errors = vsi_stats->rx_length_errors; + + return stats; } /** @@ -376,10 +415,14 @@ void i40e_vsi_reset_stats(struct i40e_vsi *vsi) memset(&vsi->eth_stats_offsets, 0, sizeof(vsi->eth_stats_offsets)); if (vsi->rx_rings) for (i = 0; i < vsi->num_queue_pairs; i++) { - memset(&vsi->rx_rings[i].rx_stats, 0 , - sizeof(vsi->rx_rings[i].rx_stats)); - memset(&vsi->tx_rings[i].tx_stats, 0, - sizeof(vsi->tx_rings[i].tx_stats)); + memset(&vsi->rx_rings[i]->stats, 0 , + sizeof(vsi->rx_rings[i]->stats)); + memset(&vsi->rx_rings[i]->rx_stats, 0 , + sizeof(vsi->rx_rings[i]->rx_stats)); + memset(&vsi->tx_rings[i]->stats, 0 , + sizeof(vsi->tx_rings[i]->stats)); + memset(&vsi->tx_rings[i]->tx_stats, 0, + sizeof(vsi->tx_rings[i]->tx_stats)); } vsi->stat_offsets_loaded = false; } @@ -598,7 +641,7 @@ static void i40e_update_link_xoff_rx(struct i40e_pf *pf) continue; for (i = 0; i < vsi->num_queue_pairs; i++) { - struct i40e_ring *ring = &vsi->tx_rings[i]; + struct i40e_ring *ring = vsi->tx_rings[i]; clear_bit(__I40E_HANG_CHECK_ARMED, &ring->state); } } @@ -652,7 +695,7 @@ static void i40e_update_prio_xoff_rx(struct i40e_pf *pf) continue; for (i = 0; i < vsi->num_queue_pairs; i++) { - struct i40e_ring *ring = &vsi->tx_rings[i]; + struct i40e_ring *ring = vsi->tx_rings[i]; tc = ring->dcb_tc; if (xoff[tc]) @@ -704,21 +747,38 @@ void i40e_update_stats(struct i40e_vsi *vsi) tx_restart = tx_busy = 0; rx_page = 0; rx_buf = 0; + rcu_read_lock(); for (q = 0; q < vsi->num_queue_pairs; q++) { struct i40e_ring *p; + u64 bytes, packets; + unsigned int start; - p = &vsi->rx_rings[q]; - rx_b += p->rx_stats.bytes; - rx_p += p->rx_stats.packets; - rx_buf += p->rx_stats.alloc_rx_buff_failed; - rx_page += p->rx_stats.alloc_rx_page_failed; + /* locate Tx ring */ + p = ACCESS_ONCE(vsi->tx_rings[q]); - p = &vsi->tx_rings[q]; - tx_b += p->tx_stats.bytes; - tx_p += p->tx_stats.packets; + do { + start = u64_stats_fetch_begin_bh(&p->syncp); + packets = p->stats.packets; + bytes = p->stats.bytes; + } while (u64_stats_fetch_retry_bh(&p->syncp, start)); + tx_b += bytes; + tx_p += packets; tx_restart += p->tx_stats.restart_queue; tx_busy += p->tx_stats.tx_busy; + + /* Rx queue is part of the same block as Tx queue */ + p = &p[1]; + do { + start = u64_stats_fetch_begin_bh(&p->syncp); + packets = p->stats.packets; + bytes = p->stats.bytes; + } while (u64_stats_fetch_retry_bh(&p->syncp, start)); + rx_b += bytes; + rx_p += packets; + rx_buf += p->rx_stats.alloc_rx_buff_failed; + rx_page += p->rx_stats.alloc_rx_page_failed; } + rcu_read_unlock(); vsi->tx_restart = tx_restart; vsi->tx_busy = tx_busy; vsi->rx_page_failed = rx_page; @@ -1988,7 +2048,7 @@ static int i40e_vsi_setup_tx_resources(struct i40e_vsi *vsi) int i, err = 0; for (i = 0; i < vsi->num_queue_pairs && !err; i++) - err = i40e_setup_tx_descriptors(&vsi->tx_rings[i]); + err = i40e_setup_tx_descriptors(vsi->tx_rings[i]); return err; } @@ -2004,8 +2064,8 @@ static void i40e_vsi_free_tx_resources(struct i40e_vsi *vsi) int i; for (i = 0; i < vsi->num_queue_pairs; i++) - if (vsi->tx_rings[i].desc) - i40e_free_tx_resources(&vsi->tx_rings[i]); + if (vsi->tx_rings[i]->desc) + i40e_free_tx_resources(vsi->tx_rings[i]); } /** @@ -2023,7 +2083,7 @@ static int i40e_vsi_setup_rx_resources(struct i40e_vsi *vsi) int i, err = 0; for (i = 0; i < vsi->num_queue_pairs && !err; i++) - err = i40e_setup_rx_descriptors(&vsi->rx_rings[i]); + err = i40e_setup_rx_descriptors(vsi->rx_rings[i]); return err; } @@ -2038,8 +2098,8 @@ static void i40e_vsi_free_rx_resources(struct i40e_vsi *vsi) int i; for (i = 0; i < vsi->num_queue_pairs; i++) - if (vsi->rx_rings[i].desc) - i40e_free_rx_resources(&vsi->rx_rings[i]); + if (vsi->rx_rings[i]->desc) + i40e_free_rx_resources(vsi->rx_rings[i]); } /** @@ -2114,8 +2174,8 @@ static int i40e_configure_tx_ring(struct i40e_ring *ring) /* Now associate this queue with this PCI function */ qtx_ctl = I40E_QTX_CTL_PF_QUEUE; - qtx_ctl |= ((hw->hmc.hmc_fn_id << I40E_QTX_CTL_PF_INDX_SHIFT) - & I40E_QTX_CTL_PF_INDX_MASK); + qtx_ctl |= ((hw->pf_id << I40E_QTX_CTL_PF_INDX_SHIFT) & + I40E_QTX_CTL_PF_INDX_MASK); wr32(hw, I40E_QTX_CTL(pf_q), qtx_ctl); i40e_flush(hw); @@ -2223,8 +2283,8 @@ static int i40e_vsi_configure_tx(struct i40e_vsi *vsi) int err = 0; u16 i; - for (i = 0; (i < vsi->num_queue_pairs) && (!err); i++) - err = i40e_configure_tx_ring(&vsi->tx_rings[i]); + for (i = 0; (i < vsi->num_queue_pairs) && !err; i++) + err = i40e_configure_tx_ring(vsi->tx_rings[i]); return err; } @@ -2274,7 +2334,7 @@ static int i40e_vsi_configure_rx(struct i40e_vsi *vsi) /* set up individual rings */ for (i = 0; i < vsi->num_queue_pairs && !err; i++) - err = i40e_configure_rx_ring(&vsi->rx_rings[i]); + err = i40e_configure_rx_ring(vsi->rx_rings[i]); return err; } @@ -2298,8 +2358,8 @@ static void i40e_vsi_config_dcb_rings(struct i40e_vsi *vsi) qoffset = vsi->tc_config.tc_info[n].qoffset; qcount = vsi->tc_config.tc_info[n].qcount; for (i = qoffset; i < (qoffset + qcount); i++) { - struct i40e_ring *rx_ring = &vsi->rx_rings[i]; - struct i40e_ring *tx_ring = &vsi->tx_rings[i]; + struct i40e_ring *rx_ring = vsi->rx_rings[i]; + struct i40e_ring *tx_ring = vsi->tx_rings[i]; rx_ring->dcb_tc = n; tx_ring->dcb_tc = n; } @@ -2354,8 +2414,8 @@ static void i40e_vsi_configure_msix(struct i40e_vsi *vsi) */ qp = vsi->base_queue; vector = vsi->base_vector; - q_vector = vsi->q_vectors; - for (i = 0; i < vsi->num_q_vectors; i++, q_vector++, vector++) { + for (i = 0; i < vsi->num_q_vectors; i++, vector++) { + q_vector = vsi->q_vectors[i]; q_vector->rx.itr = ITR_TO_REG(vsi->rx_itr_setting); q_vector->rx.latency_range = I40E_LOW_LATENCY; wr32(hw, I40E_PFINT_ITRN(I40E_RX_ITR, vector - 1), @@ -2435,7 +2495,7 @@ static void i40e_enable_misc_int_causes(struct i40e_hw *hw) **/ static void i40e_configure_msi_and_legacy(struct i40e_vsi *vsi) { - struct i40e_q_vector *q_vector = vsi->q_vectors; + struct i40e_q_vector *q_vector = vsi->q_vectors[0]; struct i40e_pf *pf = vsi->back; struct i40e_hw *hw = &pf->hw; u32 val; @@ -2472,7 +2532,7 @@ static void i40e_configure_msi_and_legacy(struct i40e_vsi *vsi) * i40e_irq_dynamic_enable_icr0 - Enable default interrupt generation for icr0 * @pf: board private structure **/ -static void i40e_irq_dynamic_enable_icr0(struct i40e_pf *pf) +void i40e_irq_dynamic_enable_icr0(struct i40e_pf *pf) { struct i40e_hw *hw = &pf->hw; u32 val; @@ -2500,7 +2560,7 @@ void i40e_irq_dynamic_enable(struct i40e_vsi *vsi, int vector) I40E_PFINT_DYN_CTLN_CLEARPBA_MASK | (I40E_ITR_NONE << I40E_PFINT_DYN_CTLN_ITR_INDX_SHIFT); wr32(hw, I40E_PFINT_DYN_CTLN(vector - 1), val); - i40e_flush(hw); + /* skip the flush */ } /** @@ -2512,7 +2572,7 @@ static irqreturn_t i40e_msix_clean_rings(int irq, void *data) { struct i40e_q_vector *q_vector = data; - if (!q_vector->tx.ring[0] && !q_vector->rx.ring[0]) + if (!q_vector->tx.ring && !q_vector->rx.ring) return IRQ_HANDLED; napi_schedule(&q_vector->napi); @@ -2529,7 +2589,7 @@ static irqreturn_t i40e_fdir_clean_rings(int irq, void *data) { struct i40e_q_vector *q_vector = data; - if (!q_vector->tx.ring[0] && !q_vector->rx.ring[0]) + if (!q_vector->tx.ring && !q_vector->rx.ring) return IRQ_HANDLED; pr_info("fdir ring cleaning needed\n"); @@ -2554,16 +2614,16 @@ static int i40e_vsi_request_irq_msix(struct i40e_vsi *vsi, char *basename) int vector, err; for (vector = 0; vector < q_vectors; vector++) { - struct i40e_q_vector *q_vector = &(vsi->q_vectors[vector]); + struct i40e_q_vector *q_vector = vsi->q_vectors[vector]; - if (q_vector->tx.ring[0] && q_vector->rx.ring[0]) { + if (q_vector->tx.ring && q_vector->rx.ring) { snprintf(q_vector->name, sizeof(q_vector->name) - 1, "%s-%s-%d", basename, "TxRx", rx_int_idx++); tx_int_idx++; - } else if (q_vector->rx.ring[0]) { + } else if (q_vector->rx.ring) { snprintf(q_vector->name, sizeof(q_vector->name) - 1, "%s-%s-%d", basename, "rx", rx_int_idx++); - } else if (q_vector->tx.ring[0]) { + } else if (q_vector->tx.ring) { snprintf(q_vector->name, sizeof(q_vector->name) - 1, "%s-%s-%d", basename, "tx", tx_int_idx++); } else { @@ -2611,8 +2671,8 @@ static void i40e_vsi_disable_irq(struct i40e_vsi *vsi) int i; for (i = 0; i < vsi->num_queue_pairs; i++) { - wr32(hw, I40E_QINT_TQCTL(vsi->tx_rings[i].reg_idx), 0); - wr32(hw, I40E_QINT_RQCTL(vsi->rx_rings[i].reg_idx), 0); + wr32(hw, I40E_QINT_TQCTL(vsi->tx_rings[i]->reg_idx), 0); + wr32(hw, I40E_QINT_RQCTL(vsi->rx_rings[i]->reg_idx), 0); } if (pf->flags & I40E_FLAG_MSIX_ENABLED) { @@ -2649,6 +2709,7 @@ static int i40e_vsi_enable_irq(struct i40e_vsi *vsi) i40e_irq_dynamic_enable_icr0(pf); } + i40e_flush(&pf->hw); return 0; } @@ -2681,14 +2742,14 @@ static irqreturn_t i40e_intr(int irq, void *data) icr0 = rd32(hw, I40E_PFINT_ICR0); - /* if sharing a legacy IRQ, we might get called w/o an intr pending */ - if ((icr0 & I40E_PFINT_ICR0_INTEVENT_MASK) == 0) - return IRQ_NONE; - val = rd32(hw, I40E_PFINT_DYN_CTL0); val = val | I40E_PFINT_DYN_CTL0_CLEARPBA_MASK; wr32(hw, I40E_PFINT_DYN_CTL0, val); + /* if sharing a legacy IRQ, we might get called w/o an intr pending */ + if ((icr0 & I40E_PFINT_ICR0_INTEVENT_MASK) == 0) + return IRQ_NONE; + ena_mask = rd32(hw, I40E_PFINT_ICR0_ENA); /* only q0 is used in MSI/Legacy mode, and none are used in MSIX */ @@ -2702,10 +2763,9 @@ static irqreturn_t i40e_intr(int irq, void *data) qval = rd32(hw, I40E_QINT_TQCTL(0)); qval &= ~I40E_QINT_TQCTL_CAUSE_ENA_MASK; wr32(hw, I40E_QINT_TQCTL(0), qval); - i40e_flush(hw); if (!test_bit(__I40E_DOWN, &pf->state)) - napi_schedule(&pf->vsi[pf->lan_vsi]->q_vectors[0].napi); + napi_schedule(&pf->vsi[pf->lan_vsi]->q_vectors[0]->napi); } if (icr0 & I40E_PFINT_ICR0_ADMINQ_MASK) { @@ -2764,7 +2824,6 @@ static irqreturn_t i40e_intr(int irq, void *data) /* re-enable interrupt causes */ wr32(hw, I40E_PFINT_ICR0_ENA, ena_mask); - i40e_flush(hw); if (!test_bit(__I40E_DOWN, &pf->state)) { i40e_service_event_schedule(pf); i40e_irq_dynamic_enable_icr0(pf); @@ -2774,40 +2833,26 @@ static irqreturn_t i40e_intr(int irq, void *data) } /** - * i40e_map_vector_to_rxq - Assigns the Rx queue to the vector + * i40e_map_vector_to_qp - Assigns the queue pair to the vector * @vsi: the VSI being configured * @v_idx: vector index - * @r_idx: rx queue index + * @qp_idx: queue pair index **/ -static void map_vector_to_rxq(struct i40e_vsi *vsi, int v_idx, int r_idx) +static void map_vector_to_qp(struct i40e_vsi *vsi, int v_idx, int qp_idx) { - struct i40e_q_vector *q_vector = &(vsi->q_vectors[v_idx]); - struct i40e_ring *rx_ring = &(vsi->rx_rings[r_idx]); - - rx_ring->q_vector = q_vector; - q_vector->rx.ring[q_vector->rx.count] = rx_ring; - q_vector->rx.count++; - q_vector->rx.latency_range = I40E_LOW_LATENCY; - q_vector->vsi = vsi; -} - -/** - * i40e_map_vector_to_txq - Assigns the Tx queue to the vector - * @vsi: the VSI being configured - * @v_idx: vector index - * @t_idx: tx queue index - **/ -static void map_vector_to_txq(struct i40e_vsi *vsi, int v_idx, int t_idx) -{ - struct i40e_q_vector *q_vector = &(vsi->q_vectors[v_idx]); - struct i40e_ring *tx_ring = &(vsi->tx_rings[t_idx]); + struct i40e_q_vector *q_vector = vsi->q_vectors[v_idx]; + struct i40e_ring *tx_ring = vsi->tx_rings[qp_idx]; + struct i40e_ring *rx_ring = vsi->rx_rings[qp_idx]; tx_ring->q_vector = q_vector; - q_vector->tx.ring[q_vector->tx.count] = tx_ring; + tx_ring->next = q_vector->tx.ring; + q_vector->tx.ring = tx_ring; q_vector->tx.count++; - q_vector->tx.latency_range = I40E_LOW_LATENCY; - q_vector->num_ringpairs++; - q_vector->vsi = vsi; + + rx_ring->q_vector = q_vector; + rx_ring->next = q_vector->rx.ring; + q_vector->rx.ring = rx_ring; + q_vector->rx.count++; } /** @@ -2823,7 +2868,7 @@ static void i40e_vsi_map_rings_to_vectors(struct i40e_vsi *vsi) { int qp_remaining = vsi->num_queue_pairs; int q_vectors = vsi->num_q_vectors; - int qp_per_vector; + int num_ringpairs; int v_start = 0; int qp_idx = 0; @@ -2831,11 +2876,21 @@ static void i40e_vsi_map_rings_to_vectors(struct i40e_vsi *vsi) * group them so there are multiple queues per vector. */ for (; v_start < q_vectors && qp_remaining; v_start++) { - qp_per_vector = DIV_ROUND_UP(qp_remaining, q_vectors - v_start); - for (; qp_per_vector; - qp_per_vector--, qp_idx++, qp_remaining--) { - map_vector_to_rxq(vsi, v_start, qp_idx); - map_vector_to_txq(vsi, v_start, qp_idx); + struct i40e_q_vector *q_vector = vsi->q_vectors[v_start]; + + num_ringpairs = DIV_ROUND_UP(qp_remaining, q_vectors - v_start); + + q_vector->num_ringpairs = num_ringpairs; + + q_vector->rx.count = 0; + q_vector->tx.count = 0; + q_vector->rx.ring = NULL; + q_vector->tx.ring = NULL; + + while (num_ringpairs--) { + map_vector_to_qp(vsi, v_start, qp_idx); + qp_idx++; + qp_remaining--; } } } @@ -2887,7 +2942,7 @@ static void i40e_netpoll(struct net_device *netdev) pf->flags |= I40E_FLAG_IN_NETPOLL; if (pf->flags & I40E_FLAG_MSIX_ENABLED) { for (i = 0; i < vsi->num_q_vectors; i++) - i40e_msix_clean_rings(0, &vsi->q_vectors[i]); + i40e_msix_clean_rings(0, vsi->q_vectors[i]); } else { i40e_intr(pf->pdev->irq, netdev); } @@ -3073,14 +3128,14 @@ static void i40e_vsi_free_irq(struct i40e_vsi *vsi) u16 vector = i + base; /* free only the irqs that were actually requested */ - if (vsi->q_vectors[i].num_ringpairs == 0) + if (vsi->q_vectors[i]->num_ringpairs == 0) continue; /* clear the affinity_mask in the IRQ descriptor */ irq_set_affinity_hint(pf->msix_entries[vector].vector, NULL); free_irq(pf->msix_entries[vector].vector, - &vsi->q_vectors[i]); + vsi->q_vectors[i]); /* Tear down the interrupt queue link list * @@ -3164,6 +3219,39 @@ static void i40e_vsi_free_irq(struct i40e_vsi *vsi) } /** + * i40e_free_q_vector - Free memory allocated for specific interrupt vector + * @vsi: the VSI being configured + * @v_idx: Index of vector to be freed + * + * This function frees the memory allocated to the q_vector. In addition if + * NAPI is enabled it will delete any references to the NAPI struct prior + * to freeing the q_vector. + **/ +static void i40e_free_q_vector(struct i40e_vsi *vsi, int v_idx) +{ + struct i40e_q_vector *q_vector = vsi->q_vectors[v_idx]; + struct i40e_ring *ring; + + if (!q_vector) + return; + + /* disassociate q_vector from rings */ + i40e_for_each_ring(ring, q_vector->tx) + ring->q_vector = NULL; + + i40e_for_each_ring(ring, q_vector->rx) + ring->q_vector = NULL; + + /* only VSI w/ an associated netdev is set up w/ NAPI */ + if (vsi->netdev) + netif_napi_del(&q_vector->napi); + + vsi->q_vectors[v_idx] = NULL; + + kfree_rcu(q_vector, rcu); +} + +/** * i40e_vsi_free_q_vectors - Free memory allocated for interrupt vectors * @vsi: the VSI being un-configured * @@ -3174,24 +3262,8 @@ static void i40e_vsi_free_q_vectors(struct i40e_vsi *vsi) { int v_idx; - for (v_idx = 0; v_idx < vsi->num_q_vectors; v_idx++) { - struct i40e_q_vector *q_vector = &vsi->q_vectors[v_idx]; - int r_idx; - - if (!q_vector) - continue; - - /* disassociate q_vector from rings */ - for (r_idx = 0; r_idx < q_vector->tx.count; r_idx++) - q_vector->tx.ring[r_idx]->q_vector = NULL; - for (r_idx = 0; r_idx < q_vector->rx.count; r_idx++) - q_vector->rx.ring[r_idx]->q_vector = NULL; - - /* only VSI w/ an associated netdev is set up w/ NAPI */ - if (vsi->netdev) - netif_napi_del(&q_vector->napi); - } - kfree(vsi->q_vectors); + for (v_idx = 0; v_idx < vsi->num_q_vectors; v_idx++) + i40e_free_q_vector(vsi, v_idx); } /** @@ -3241,7 +3313,7 @@ static void i40e_napi_enable_all(struct i40e_vsi *vsi) return; for (q_idx = 0; q_idx < vsi->num_q_vectors; q_idx++) - napi_enable(&vsi->q_vectors[q_idx].napi); + napi_enable(&vsi->q_vectors[q_idx]->napi); } /** @@ -3256,7 +3328,7 @@ static void i40e_napi_disable_all(struct i40e_vsi *vsi) return; for (q_idx = 0; q_idx < vsi->num_q_vectors; q_idx++) - napi_disable(&vsi->q_vectors[q_idx].napi); + napi_disable(&vsi->q_vectors[q_idx]->napi); } /** @@ -3703,8 +3775,11 @@ static int i40e_up_complete(struct i40e_vsi *vsi) if ((pf->hw.phy.link_info.link_info & I40E_AQ_LINK_UP) && (vsi->netdev)) { + netdev_info(vsi->netdev, "NIC Link is Up\n"); netif_tx_start_all_queues(vsi->netdev); netif_carrier_on(vsi->netdev); + } else if (vsi->netdev) { + netdev_info(vsi->netdev, "NIC Link is Down\n"); } i40e_service_event_schedule(pf); @@ -3772,8 +3847,8 @@ void i40e_down(struct i40e_vsi *vsi) i40e_napi_disable_all(vsi); for (i = 0; i < vsi->num_queue_pairs; i++) { - i40e_clean_tx_ring(&vsi->tx_rings[i]); - i40e_clean_rx_ring(&vsi->rx_rings[i]); + i40e_clean_tx_ring(vsi->tx_rings[i]); + i40e_clean_rx_ring(vsi->rx_rings[i]); } } @@ -4153,8 +4228,9 @@ static void i40e_link_event(struct i40e_pf *pf) if (new_link == old_link) return; - netdev_info(pf->vsi[pf->lan_vsi]->netdev, - "NIC Link is %s\n", (new_link ? "Up" : "Down")); + if (!test_bit(__I40E_DOWN, &pf->vsi[pf->lan_vsi]->state)) + netdev_info(pf->vsi[pf->lan_vsi]->netdev, + "NIC Link is %s\n", (new_link ? "Up" : "Down")); /* Notify the base of the switch tree connected to * the link. Floating VEBs are not notified. @@ -4199,9 +4275,9 @@ static void i40e_check_hang_subtask(struct i40e_pf *pf) continue; for (i = 0; i < vsi->num_queue_pairs; i++) { - set_check_for_tx_hang(&vsi->tx_rings[i]); + set_check_for_tx_hang(vsi->tx_rings[i]); if (test_bit(__I40E_HANG_CHECK_ARMED, - &vsi->tx_rings[i].state)) + &vsi->tx_rings[i]->state)) armed++; } @@ -4537,7 +4613,8 @@ static void i40e_fdir_setup(struct i40e_pf *pf) bool new_vsi = false; int err, i; - if (!(pf->flags & (I40E_FLAG_FDIR_ENABLED|I40E_FLAG_FDIR_ATR_ENABLED))) + if (!(pf->flags & (I40E_FLAG_FDIR_ENABLED | + I40E_FLAG_FDIR_ATR_ENABLED))) return; pf->atr_sample_rate = I40E_DEFAULT_ATR_SAMPLE_RATE; @@ -4937,6 +5014,8 @@ static int i40e_vsi_mem_alloc(struct i40e_pf *pf, enum i40e_vsi_type type) { int ret = -ENODEV; struct i40e_vsi *vsi; + int sz_vectors; + int sz_rings; int vsi_idx; int i; @@ -4962,14 +5041,14 @@ static int i40e_vsi_mem_alloc(struct i40e_pf *pf, enum i40e_vsi_type type) vsi_idx = i; /* Found one! */ } else { ret = -ENODEV; - goto err_alloc_vsi; /* out of VSI slots! */ + goto unlock_pf; /* out of VSI slots! */ } pf->next_vsi = ++i; vsi = kzalloc(sizeof(*vsi), GFP_KERNEL); if (!vsi) { ret = -ENOMEM; - goto err_alloc_vsi; + goto unlock_pf; } vsi->type = type; vsi->back = pf; @@ -4982,14 +5061,40 @@ static int i40e_vsi_mem_alloc(struct i40e_pf *pf, enum i40e_vsi_type type) vsi->work_limit = I40E_DEFAULT_IRQ_WORK; INIT_LIST_HEAD(&vsi->mac_filter_list); - i40e_set_num_rings_in_vsi(vsi); + ret = i40e_set_num_rings_in_vsi(vsi); + if (ret) + goto err_rings; + + /* allocate memory for ring pointers */ + sz_rings = sizeof(struct i40e_ring *) * vsi->alloc_queue_pairs * 2; + vsi->tx_rings = kzalloc(sz_rings, GFP_KERNEL); + if (!vsi->tx_rings) { + ret = -ENOMEM; + goto err_rings; + } + vsi->rx_rings = &vsi->tx_rings[vsi->alloc_queue_pairs]; + + /* allocate memory for q_vector pointers */ + sz_vectors = sizeof(struct i40e_q_vectors *) * vsi->num_q_vectors; + vsi->q_vectors = kzalloc(sz_vectors, GFP_KERNEL); + if (!vsi->q_vectors) { + ret = -ENOMEM; + goto err_vectors; + } /* Setup default MSIX irq handler for VSI */ i40e_vsi_setup_irqhandler(vsi, i40e_msix_clean_rings); pf->vsi[vsi_idx] = vsi; ret = vsi_idx; -err_alloc_vsi: + goto unlock_pf; + +err_vectors: + kfree(vsi->tx_rings); +err_rings: + pf->next_vsi = i - 1; + kfree(vsi); +unlock_pf: mutex_unlock(&pf->switch_mutex); return ret; } @@ -5030,6 +5135,10 @@ static int i40e_vsi_clear(struct i40e_vsi *vsi) i40e_put_lump(pf->qp_pile, vsi->base_queue, vsi->idx); i40e_put_lump(pf->irq_pile, vsi->base_vector, vsi->idx); + /* free the ring and vector containers */ + kfree(vsi->q_vectors); + kfree(vsi->tx_rings); + pf->vsi[vsi->idx] = NULL; if (vsi->idx < pf->next_vsi) pf->next_vsi = vsi->idx; @@ -5043,34 +5152,40 @@ free_vsi: } /** + * i40e_vsi_clear_rings - Deallocates the Rx and Tx rings for the provided VSI + * @vsi: the VSI being cleaned + **/ +static s32 i40e_vsi_clear_rings(struct i40e_vsi *vsi) +{ + int i; + + if (vsi->tx_rings[0]) + for (i = 0; i < vsi->alloc_queue_pairs; i++) { + kfree_rcu(vsi->tx_rings[i], rcu); + vsi->tx_rings[i] = NULL; + vsi->rx_rings[i] = NULL; + } + + return 0; +} + +/** * i40e_alloc_rings - Allocates the Rx and Tx rings for the provided VSI * @vsi: the VSI being configured **/ static int i40e_alloc_rings(struct i40e_vsi *vsi) { struct i40e_pf *pf = vsi->back; - int ret = 0; int i; - vsi->rx_rings = kcalloc(vsi->alloc_queue_pairs, - sizeof(struct i40e_ring), GFP_KERNEL); - if (!vsi->rx_rings) { - ret = -ENOMEM; - goto err_alloc_rings; - } - - vsi->tx_rings = kcalloc(vsi->alloc_queue_pairs, - sizeof(struct i40e_ring), GFP_KERNEL); - if (!vsi->tx_rings) { - ret = -ENOMEM; - kfree(vsi->rx_rings); - goto err_alloc_rings; - } - /* Set basic values in the rings to be used later during open() */ for (i = 0; i < vsi->alloc_queue_pairs; i++) { - struct i40e_ring *rx_ring = &vsi->rx_rings[i]; - struct i40e_ring *tx_ring = &vsi->tx_rings[i]; + struct i40e_ring *tx_ring; + struct i40e_ring *rx_ring; + + tx_ring = kzalloc(sizeof(struct i40e_ring) * 2, GFP_KERNEL); + if (!tx_ring) + goto err_out; tx_ring->queue_index = i; tx_ring->reg_idx = vsi->base_queue + i; @@ -5081,7 +5196,9 @@ static int i40e_alloc_rings(struct i40e_vsi *vsi) tx_ring->count = vsi->num_desc; tx_ring->size = 0; tx_ring->dcb_tc = 0; + vsi->tx_rings[i] = tx_ring; + rx_ring = &tx_ring[1]; rx_ring->queue_index = i; rx_ring->reg_idx = vsi->base_queue + i; rx_ring->ring_active = false; @@ -5095,24 +5212,14 @@ static int i40e_alloc_rings(struct i40e_vsi *vsi) set_ring_16byte_desc_enabled(rx_ring); else clear_ring_16byte_desc_enabled(rx_ring); - } - -err_alloc_rings: - return ret; -} - -/** - * i40e_vsi_clear_rings - Deallocates the Rx and Tx rings for the provided VSI - * @vsi: the VSI being cleaned - **/ -static int i40e_vsi_clear_rings(struct i40e_vsi *vsi) -{ - if (vsi) { - kfree(vsi->rx_rings); - kfree(vsi->tx_rings); + vsi->rx_rings[i] = rx_ring; } return 0; + +err_out: + i40e_vsi_clear_rings(vsi); + return -ENOMEM; } /** @@ -5249,6 +5356,38 @@ static int i40e_init_msix(struct i40e_pf *pf) } /** + * i40e_alloc_q_vector - Allocate memory for a single interrupt vector + * @vsi: the VSI being configured + * @v_idx: index of the vector in the vsi struct + * + * We allocate one q_vector. If allocation fails we return -ENOMEM. + **/ +static int i40e_alloc_q_vector(struct i40e_vsi *vsi, int v_idx) +{ + struct i40e_q_vector *q_vector; + + /* allocate q_vector */ + q_vector = kzalloc(sizeof(struct i40e_q_vector), GFP_KERNEL); + if (!q_vector) + return -ENOMEM; + + q_vector->vsi = vsi; + q_vector->v_idx = v_idx; + cpumask_set_cpu(v_idx, &q_vector->affinity_mask); + if (vsi->netdev) + netif_napi_add(vsi->netdev, &q_vector->napi, + i40e_napi_poll, vsi->work_limit); + + q_vector->rx.latency_range = I40E_LOW_LATENCY; + q_vector->tx.latency_range = I40E_LOW_LATENCY; + + /* tie q_vector and vsi together */ + vsi->q_vectors[v_idx] = q_vector; + + return 0; +} + +/** * i40e_alloc_q_vectors - Allocate memory for interrupt vectors * @vsi: the VSI being configured * @@ -5259,6 +5398,7 @@ static int i40e_alloc_q_vectors(struct i40e_vsi *vsi) { struct i40e_pf *pf = vsi->back; int v_idx, num_q_vectors; + int err; /* if not MSIX, give the one vector only to the LAN VSI */ if (pf->flags & I40E_FLAG_MSIX_ENABLED) @@ -5268,22 +5408,19 @@ static int i40e_alloc_q_vectors(struct i40e_vsi *vsi) else return -EINVAL; - vsi->q_vectors = kcalloc(num_q_vectors, - sizeof(struct i40e_q_vector), - GFP_KERNEL); - if (!vsi->q_vectors) - return -ENOMEM; - for (v_idx = 0; v_idx < num_q_vectors; v_idx++) { - vsi->q_vectors[v_idx].vsi = vsi; - vsi->q_vectors[v_idx].v_idx = v_idx; - cpumask_set_cpu(v_idx, &vsi->q_vectors[v_idx].affinity_mask); - if (vsi->netdev) - netif_napi_add(vsi->netdev, &vsi->q_vectors[v_idx].napi, - i40e_napi_poll, vsi->work_limit); + err = i40e_alloc_q_vector(vsi, v_idx); + if (err) + goto err_out; } return 0; + +err_out: + while (v_idx--) + i40e_free_q_vector(vsi, v_idx); + + return err; } /** @@ -5297,7 +5434,8 @@ static void i40e_init_interrupt_scheme(struct i40e_pf *pf) if (pf->flags & I40E_FLAG_MSIX_ENABLED) { err = i40e_init_msix(pf); if (err) { - pf->flags &= ~(I40E_FLAG_RSS_ENABLED | + pf->flags &= ~(I40E_FLAG_MSIX_ENABLED | + I40E_FLAG_RSS_ENABLED | I40E_FLAG_MQ_ENABLED | I40E_FLAG_DCB_ENABLED | I40E_FLAG_SRIOV_ENABLED | @@ -5312,14 +5450,17 @@ static void i40e_init_interrupt_scheme(struct i40e_pf *pf) if (!(pf->flags & I40E_FLAG_MSIX_ENABLED) && (pf->flags & I40E_FLAG_MSI_ENABLED)) { + dev_info(&pf->pdev->dev, "MSIX not available, trying MSI\n"); err = pci_enable_msi(pf->pdev); if (err) { - dev_info(&pf->pdev->dev, - "MSI init failed (%d), trying legacy.\n", err); + dev_info(&pf->pdev->dev, "MSI init failed - %d\n", err); pf->flags &= ~I40E_FLAG_MSI_ENABLED; } } + if (!(pf->flags & (I40E_FLAG_MSIX_ENABLED | I40E_FLAG_MSI_ENABLED))) + dev_info(&pf->pdev->dev, "MSIX and MSI not available, falling back to Legacy IRQ\n"); + /* track first vector for misc interrupts */ err = i40e_get_lump(pf, pf->irq_pile, 1, I40E_PILE_VALID_BIT-1); } @@ -5950,7 +6091,7 @@ static int i40e_vsi_setup_vectors(struct i40e_vsi *vsi) int ret = -ENOENT; struct i40e_pf *pf = vsi->back; - if (vsi->q_vectors) { + if (vsi->q_vectors[0]) { dev_info(&pf->pdev->dev, "VSI %d has existing q_vectors\n", vsi->seid); return -EEXIST; @@ -5972,8 +6113,9 @@ static int i40e_vsi_setup_vectors(struct i40e_vsi *vsi) goto vector_setup_out; } - vsi->base_vector = i40e_get_lump(pf, pf->irq_pile, - vsi->num_q_vectors, vsi->idx); + if (vsi->num_q_vectors) + vsi->base_vector = i40e_get_lump(pf, pf->irq_pile, + vsi->num_q_vectors, vsi->idx); if (vsi->base_vector < 0) { dev_info(&pf->pdev->dev, "failed to get q tracking for VSI %d, err=%d\n", @@ -7062,8 +7204,10 @@ static int i40e_probe(struct pci_dev *pdev, const struct pci_device_id *ent) */ len = sizeof(struct i40e_vsi *) * pf->hw.func_caps.num_vsis; pf->vsi = kzalloc(len, GFP_KERNEL); - if (!pf->vsi) + if (!pf->vsi) { + err = -ENOMEM; goto err_switch_setup; + } err = i40e_setup_pf_switch(pf); if (err) { diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c index 49d2cfa..f1f03bc 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c @@ -37,6 +37,7 @@ static inline __le64 build_ctob(u32 td_cmd, u32 td_offset, unsigned int size, ((u64)td_tag << I40E_TXD_QW1_L2TAG1_SHIFT)); } +#define I40E_TXD_CMD (I40E_TX_DESC_CMD_EOP | I40E_TX_DESC_CMD_RS) /** * i40e_program_fdir_filter - Program a Flow Director filter * @fdir_input: Packet data that will be filter parameters @@ -50,6 +51,7 @@ int i40e_program_fdir_filter(struct i40e_fdir_data *fdir_data, struct i40e_tx_buffer *tx_buf; struct i40e_tx_desc *tx_desc; struct i40e_ring *tx_ring; + unsigned int fpt, dcc; struct i40e_vsi *vsi; struct device *dev; dma_addr_t dma; @@ -64,93 +66,78 @@ int i40e_program_fdir_filter(struct i40e_fdir_data *fdir_data, if (!vsi) return -ENOENT; - tx_ring = &vsi->tx_rings[0]; + tx_ring = vsi->tx_rings[0]; dev = tx_ring->dev; dma = dma_map_single(dev, fdir_data->raw_packet, - I40E_FDIR_MAX_RAW_PACKET_LOOKUP, DMA_TO_DEVICE); + I40E_FDIR_MAX_RAW_PACKET_LOOKUP, DMA_TO_DEVICE); if (dma_mapping_error(dev, dma)) goto dma_fail; /* grab the next descriptor */ - fdir_desc = I40E_TX_FDIRDESC(tx_ring, tx_ring->next_to_use); - tx_buf = &tx_ring->tx_bi[tx_ring->next_to_use]; - tx_ring->next_to_use++; - if (tx_ring->next_to_use == tx_ring->count) - tx_ring->next_to_use = 0; + i = tx_ring->next_to_use; + fdir_desc = I40E_TX_FDIRDESC(tx_ring, i); + tx_buf = &tx_ring->tx_bi[i]; + + tx_ring->next_to_use = (i + 1 < tx_ring->count) ? i + 1 : 0; - fdir_desc->qindex_flex_ptype_vsi = cpu_to_le32((fdir_data->q_index - << I40E_TXD_FLTR_QW0_QINDEX_SHIFT) - & I40E_TXD_FLTR_QW0_QINDEX_MASK); + fpt = (fdir_data->q_index << I40E_TXD_FLTR_QW0_QINDEX_SHIFT) & + I40E_TXD_FLTR_QW0_QINDEX_MASK; - fdir_desc->qindex_flex_ptype_vsi |= cpu_to_le32((fdir_data->flex_off - << I40E_TXD_FLTR_QW0_FLEXOFF_SHIFT) - & I40E_TXD_FLTR_QW0_FLEXOFF_MASK); + fpt |= (fdir_data->flex_off << I40E_TXD_FLTR_QW0_FLEXOFF_SHIFT) & + I40E_TXD_FLTR_QW0_FLEXOFF_MASK; - fdir_desc->qindex_flex_ptype_vsi |= cpu_to_le32((fdir_data->pctype - << I40E_TXD_FLTR_QW0_PCTYPE_SHIFT) - & I40E_TXD_FLTR_QW0_PCTYPE_MASK); + fpt |= (fdir_data->pctype << I40E_TXD_FLTR_QW0_PCTYPE_SHIFT) & + I40E_TXD_FLTR_QW0_PCTYPE_MASK; /* Use LAN VSI Id if not programmed by user */ if (fdir_data->dest_vsi == 0) - fdir_desc->qindex_flex_ptype_vsi |= - cpu_to_le32((pf->vsi[pf->lan_vsi]->id) - << I40E_TXD_FLTR_QW0_DEST_VSI_SHIFT); + fpt |= (pf->vsi[pf->lan_vsi]->id) << + I40E_TXD_FLTR_QW0_DEST_VSI_SHIFT; else - fdir_desc->qindex_flex_ptype_vsi |= - cpu_to_le32((fdir_data->dest_vsi - << I40E_TXD_FLTR_QW0_DEST_VSI_SHIFT) - & I40E_TXD_FLTR_QW0_DEST_VSI_MASK); + fpt |= ((u32)fdir_data->dest_vsi << + I40E_TXD_FLTR_QW0_DEST_VSI_SHIFT) & + I40E_TXD_FLTR_QW0_DEST_VSI_MASK; - fdir_desc->dtype_cmd_cntindex = - cpu_to_le32(I40E_TX_DESC_DTYPE_FILTER_PROG); + fdir_desc->qindex_flex_ptype_vsi = cpu_to_le32(fpt); + + dcc = I40E_TX_DESC_DTYPE_FILTER_PROG; if (add) - fdir_desc->dtype_cmd_cntindex |= cpu_to_le32( - I40E_FILTER_PROGRAM_DESC_PCMD_ADD_UPDATE - << I40E_TXD_FLTR_QW1_PCMD_SHIFT); + dcc |= I40E_FILTER_PROGRAM_DESC_PCMD_ADD_UPDATE << + I40E_TXD_FLTR_QW1_PCMD_SHIFT; else - fdir_desc->dtype_cmd_cntindex |= cpu_to_le32( - I40E_FILTER_PROGRAM_DESC_PCMD_REMOVE - << I40E_TXD_FLTR_QW1_PCMD_SHIFT); + dcc |= I40E_FILTER_PROGRAM_DESC_PCMD_REMOVE << + I40E_TXD_FLTR_QW1_PCMD_SHIFT; - fdir_desc->dtype_cmd_cntindex |= cpu_to_le32((fdir_data->dest_ctl - << I40E_TXD_FLTR_QW1_DEST_SHIFT) - & I40E_TXD_FLTR_QW1_DEST_MASK); + dcc |= (fdir_data->dest_ctl << I40E_TXD_FLTR_QW1_DEST_SHIFT) & + I40E_TXD_FLTR_QW1_DEST_MASK; - fdir_desc->dtype_cmd_cntindex |= cpu_to_le32( - (fdir_data->fd_status << I40E_TXD_FLTR_QW1_FD_STATUS_SHIFT) - & I40E_TXD_FLTR_QW1_FD_STATUS_MASK); + dcc |= (fdir_data->fd_status << I40E_TXD_FLTR_QW1_FD_STATUS_SHIFT) & + I40E_TXD_FLTR_QW1_FD_STATUS_MASK; if (fdir_data->cnt_index != 0) { - fdir_desc->dtype_cmd_cntindex |= - cpu_to_le32(I40E_TXD_FLTR_QW1_CNT_ENA_MASK); - fdir_desc->dtype_cmd_cntindex |= - cpu_to_le32((fdir_data->cnt_index - << I40E_TXD_FLTR_QW1_CNTINDEX_SHIFT) - & I40E_TXD_FLTR_QW1_CNTINDEX_MASK); + dcc |= I40E_TXD_FLTR_QW1_CNT_ENA_MASK; + dcc |= ((u32)fdir_data->cnt_index << + I40E_TXD_FLTR_QW1_CNTINDEX_SHIFT) & + I40E_TXD_FLTR_QW1_CNTINDEX_MASK; } + fdir_desc->dtype_cmd_cntindex = cpu_to_le32(dcc); fdir_desc->fd_id = cpu_to_le32(fdir_data->fd_id); /* Now program a dummy descriptor */ - tx_desc = I40E_TX_DESC(tx_ring, tx_ring->next_to_use); - tx_buf = &tx_ring->tx_bi[tx_ring->next_to_use]; - tx_ring->next_to_use++; - if (tx_ring->next_to_use == tx_ring->count) - tx_ring->next_to_use = 0; + i = tx_ring->next_to_use; + tx_desc = I40E_TX_DESC(tx_ring, i); + + tx_ring->next_to_use = (i + 1 < tx_ring->count) ? i + 1 : 0; tx_desc->buffer_addr = cpu_to_le64(dma); - td_cmd = I40E_TX_DESC_CMD_EOP | - I40E_TX_DESC_CMD_RS | - I40E_TX_DESC_CMD_DUMMY; + td_cmd = I40E_TXD_CMD | I40E_TX_DESC_CMD_DUMMY; tx_desc->cmd_type_offset_bsz = build_ctob(td_cmd, 0, I40E_FDIR_MAX_RAW_PACKET_LOOKUP, 0); - /* Mark the data descriptor to be watched */ - tx_buf->next_to_watch = tx_desc; - /* Force memory writes to complete before letting h/w * know there are new descriptors to fetch. (Only * applicable for weak-ordered memory model archs, @@ -158,6 +145,9 @@ int i40e_program_fdir_filter(struct i40e_fdir_data *fdir_data, */ wmb(); + /* Mark the data descriptor to be watched */ + tx_buf->next_to_watch = tx_desc; + writel(tx_ring->next_to_use, tx_ring->tail); return 0; @@ -188,27 +178,30 @@ static void i40e_fd_handle_status(struct i40e_ring *rx_ring, u32 qw, u8 prog_id) } /** - * i40e_unmap_tx_resource - Release a Tx buffer + * i40e_unmap_and_free_tx_resource - Release a Tx buffer * @ring: the ring that owns the buffer * @tx_buffer: the buffer to free **/ -static inline void i40e_unmap_tx_resource(struct i40e_ring *ring, - struct i40e_tx_buffer *tx_buffer) +static void i40e_unmap_and_free_tx_resource(struct i40e_ring *ring, + struct i40e_tx_buffer *tx_buffer) { - if (tx_buffer->dma) { - if (tx_buffer->tx_flags & I40E_TX_FLAGS_MAPPED_AS_PAGE) - dma_unmap_page(ring->dev, - tx_buffer->dma, - tx_buffer->length, - DMA_TO_DEVICE); - else + if (tx_buffer->skb) { + dev_kfree_skb_any(tx_buffer->skb); + if (dma_unmap_len(tx_buffer, len)) dma_unmap_single(ring->dev, - tx_buffer->dma, - tx_buffer->length, + dma_unmap_addr(tx_buffer, dma), + dma_unmap_len(tx_buffer, len), DMA_TO_DEVICE); + } else if (dma_unmap_len(tx_buffer, len)) { + dma_unmap_page(ring->dev, + dma_unmap_addr(tx_buffer, dma), + dma_unmap_len(tx_buffer, len), + DMA_TO_DEVICE); } - tx_buffer->dma = 0; - tx_buffer->time_stamp = 0; + tx_buffer->next_to_watch = NULL; + tx_buffer->skb = NULL; + dma_unmap_len_set(tx_buffer, len, 0); + /* tx_buffer must be completely set up in the transmit path */ } /** @@ -217,7 +210,6 @@ static inline void i40e_unmap_tx_resource(struct i40e_ring *ring, **/ void i40e_clean_tx_ring(struct i40e_ring *tx_ring) { - struct i40e_tx_buffer *tx_buffer; unsigned long bi_size; u16 i; @@ -226,13 +218,8 @@ void i40e_clean_tx_ring(struct i40e_ring *tx_ring) return; /* Free all the Tx ring sk_buffs */ - for (i = 0; i < tx_ring->count; i++) { - tx_buffer = &tx_ring->tx_bi[i]; - i40e_unmap_tx_resource(tx_ring, tx_buffer); - if (tx_buffer->skb) - dev_kfree_skb_any(tx_buffer->skb); - tx_buffer->skb = NULL; - } + for (i = 0; i < tx_ring->count; i++) + i40e_unmap_and_free_tx_resource(tx_ring, &tx_ring->tx_bi[i]); bi_size = sizeof(struct i40e_tx_buffer) * tx_ring->count; memset(tx_ring->tx_bi, 0, bi_size); @@ -242,6 +229,13 @@ void i40e_clean_tx_ring(struct i40e_ring *tx_ring) tx_ring->next_to_use = 0; tx_ring->next_to_clean = 0; + + if (!tx_ring->netdev) + return; + + /* cleanup Tx queue statistics */ + netdev_tx_reset_queue(netdev_get_tx_queue(tx_ring->netdev, + tx_ring->queue_index)); } /** @@ -300,14 +294,14 @@ static bool i40e_check_tx_hang(struct i40e_ring *tx_ring) * run the check_tx_hang logic with a transmit completion * pending but without time to complete it yet. */ - if ((tx_ring->tx_stats.tx_done_old == tx_ring->tx_stats.packets) && + if ((tx_ring->tx_stats.tx_done_old == tx_ring->stats.packets) && tx_pending) { /* make sure it is true for two checks in a row */ ret = test_and_set_bit(__I40E_HANG_CHECK_ARMED, &tx_ring->state); } else { /* update completed stats and disarm the hang check */ - tx_ring->tx_stats.tx_done_old = tx_ring->tx_stats.packets; + tx_ring->tx_stats.tx_done_old = tx_ring->stats.packets; clear_bit(__I40E_HANG_CHECK_ARMED, &tx_ring->state); } @@ -331,62 +325,88 @@ static bool i40e_clean_tx_irq(struct i40e_ring *tx_ring, int budget) tx_buf = &tx_ring->tx_bi[i]; tx_desc = I40E_TX_DESC(tx_ring, i); + i -= tx_ring->count; - for (; budget; budget--) { - struct i40e_tx_desc *eop_desc; - - eop_desc = tx_buf->next_to_watch; + do { + struct i40e_tx_desc *eop_desc = tx_buf->next_to_watch; /* if next_to_watch is not set then there is no work pending */ if (!eop_desc) break; + /* prevent any other reads prior to eop_desc */ + read_barrier_depends(); + /* if the descriptor isn't done, no work yet to do */ if (!(eop_desc->cmd_type_offset_bsz & cpu_to_le64(I40E_TX_DESC_DTYPE_DESC_DONE))) break; - /* count the packet as being completed */ - tx_ring->tx_stats.completed++; + /* clear next_to_watch to prevent false hangs */ tx_buf->next_to_watch = NULL; - tx_buf->time_stamp = 0; - - /* set memory barrier before eop_desc is verified */ - rmb(); - do { - i40e_unmap_tx_resource(tx_ring, tx_buf); + /* update the statistics for this packet */ + total_bytes += tx_buf->bytecount; + total_packets += tx_buf->gso_segs; - /* clear dtype status */ - tx_desc->cmd_type_offset_bsz &= - ~cpu_to_le64(I40E_TXD_QW1_DTYPE_MASK); + /* free the skb */ + dev_kfree_skb_any(tx_buf->skb); - if (likely(tx_desc == eop_desc)) { - eop_desc = NULL; + /* unmap skb header data */ + dma_unmap_single(tx_ring->dev, + dma_unmap_addr(tx_buf, dma), + dma_unmap_len(tx_buf, len), + DMA_TO_DEVICE); - dev_kfree_skb_any(tx_buf->skb); - tx_buf->skb = NULL; + /* clear tx_buffer data */ + tx_buf->skb = NULL; + dma_unmap_len_set(tx_buf, len, 0); - total_bytes += tx_buf->bytecount; - total_packets += tx_buf->gso_segs; - } + /* unmap remaining buffers */ + while (tx_desc != eop_desc) { tx_buf++; tx_desc++; i++; - if (unlikely(i == tx_ring->count)) { - i = 0; + if (unlikely(!i)) { + i -= tx_ring->count; tx_buf = tx_ring->tx_bi; tx_desc = I40E_TX_DESC(tx_ring, 0); } - } while (eop_desc); - } + /* unmap any remaining paged data */ + if (dma_unmap_len(tx_buf, len)) { + dma_unmap_page(tx_ring->dev, + dma_unmap_addr(tx_buf, dma), + dma_unmap_len(tx_buf, len), + DMA_TO_DEVICE); + dma_unmap_len_set(tx_buf, len, 0); + } + } + + /* move us one more past the eop_desc for start of next pkt */ + tx_buf++; + tx_desc++; + i++; + if (unlikely(!i)) { + i -= tx_ring->count; + tx_buf = tx_ring->tx_bi; + tx_desc = I40E_TX_DESC(tx_ring, 0); + } + + /* update budget accounting */ + budget--; + } while (likely(budget)); + + i += tx_ring->count; tx_ring->next_to_clean = i; - tx_ring->tx_stats.bytes += total_bytes; - tx_ring->tx_stats.packets += total_packets; + u64_stats_update_begin(&tx_ring->syncp); + tx_ring->stats.bytes += total_bytes; + tx_ring->stats.packets += total_packets; + u64_stats_update_end(&tx_ring->syncp); tx_ring->q_vector->tx.total_bytes += total_bytes; tx_ring->q_vector->tx.total_packets += total_packets; + if (check_for_tx_hang(tx_ring) && i40e_check_tx_hang(tx_ring)) { /* schedule immediate reset if we believe we hung */ dev_info(tx_ring->dev, "Detected Tx Unit Hang\n" @@ -414,6 +434,10 @@ static bool i40e_clean_tx_irq(struct i40e_ring *tx_ring, int budget) return true; } + netdev_tx_completed_queue(netdev_get_tx_queue(tx_ring->netdev, + tx_ring->queue_index), + total_packets, total_bytes); + #define TX_WAKE_THRESHOLD (DESC_NEEDED * 2) if (unlikely(total_packets && netif_carrier_ok(tx_ring->netdev) && (I40E_DESC_UNUSED(tx_ring) >= TX_WAKE_THRESHOLD))) { @@ -524,8 +548,6 @@ static void i40e_update_dynamic_itr(struct i40e_q_vector *q_vector) i40e_set_new_dynamic_itr(&q_vector->tx); if (old_itr != q_vector->tx.itr) wr32(hw, reg_addr, q_vector->tx.itr); - - i40e_flush(hw); } /** @@ -1042,8 +1064,10 @@ next_desc: } rx_ring->next_to_clean = i; - rx_ring->rx_stats.packets += total_rx_packets; - rx_ring->rx_stats.bytes += total_rx_bytes; + u64_stats_update_begin(&rx_ring->syncp); + rx_ring->stats.packets += total_rx_packets; + rx_ring->stats.bytes += total_rx_bytes; + u64_stats_update_end(&rx_ring->syncp); rx_ring->q_vector->rx.total_packets += total_rx_packets; rx_ring->q_vector->rx.total_bytes += total_rx_bytes; @@ -1067,27 +1091,28 @@ int i40e_napi_poll(struct napi_struct *napi, int budget) struct i40e_q_vector *q_vector = container_of(napi, struct i40e_q_vector, napi); struct i40e_vsi *vsi = q_vector->vsi; + struct i40e_ring *ring; bool clean_complete = true; int budget_per_ring; - int i; if (test_bit(__I40E_DOWN, &vsi->state)) { napi_complete(napi); return 0; } + /* Since the actual Tx work is minimal, we can give the Tx a larger + * budget and be more aggressive about cleaning up the Tx descriptors. + */ + i40e_for_each_ring(ring, q_vector->tx) + clean_complete &= i40e_clean_tx_irq(ring, vsi->work_limit); + /* We attempt to distribute budget to each Rx queue fairly, but don't * allow the budget to go below 1 because that would exit polling early. - * Since the actual Tx work is minimal, we can give the Tx a larger - * budget and be more aggressive about cleaning up the Tx descriptors. */ budget_per_ring = max(budget/q_vector->num_ringpairs, 1); - for (i = 0; i < q_vector->num_ringpairs; i++) { - clean_complete &= i40e_clean_tx_irq(q_vector->tx.ring[i], - vsi->work_limit); - clean_complete &= i40e_clean_rx_irq(q_vector->rx.ring[i], - budget_per_ring); - } + + i40e_for_each_ring(ring, q_vector->rx) + clean_complete &= i40e_clean_rx_irq(ring, budget_per_ring); /* If work not completed, return budget and polling will return */ if (!clean_complete) @@ -1117,7 +1142,8 @@ int i40e_napi_poll(struct napi_struct *napi, int budget) qval = rd32(hw, I40E_QINT_TQCTL(0)); qval |= I40E_QINT_TQCTL_CAUSE_ENA_MASK; wr32(hw, I40E_QINT_TQCTL(0), qval); - i40e_flush(hw); + + i40e_irq_dynamic_enable_icr0(vsi->back); } } @@ -1144,6 +1170,7 @@ static void i40e_atr(struct i40e_ring *tx_ring, struct sk_buff *skb, struct tcphdr *th; unsigned int hlen; u32 flex_ptype, dtype_cmd; + u16 i; /* make sure ATR is enabled */ if (!(pf->flags & I40E_FLAG_FDIR_ATR_ENABLED)) @@ -1183,10 +1210,11 @@ static void i40e_atr(struct i40e_ring *tx_ring, struct sk_buff *skb, tx_ring->atr_count = 0; /* grab the next descriptor */ - fdir_desc = I40E_TX_FDIRDESC(tx_ring, tx_ring->next_to_use); - tx_ring->next_to_use++; - if (tx_ring->next_to_use == tx_ring->count) - tx_ring->next_to_use = 0; + i = tx_ring->next_to_use; + fdir_desc = I40E_TX_FDIRDESC(tx_ring, i); + + i++; + tx_ring->next_to_use = (i < tx_ring->count) ? i : 0; flex_ptype = (tx_ring->queue_index << I40E_TXD_FLTR_QW0_QINDEX_SHIFT) & I40E_TXD_FLTR_QW0_QINDEX_MASK; @@ -1216,7 +1244,6 @@ static void i40e_atr(struct i40e_ring *tx_ring, struct sk_buff *skb, fdir_desc->dtype_cmd_cntindex = cpu_to_le32(dtype_cmd); } -#define I40E_TXD_CMD (I40E_TX_DESC_CMD_EOP | I40E_TX_DESC_CMD_RS) /** * i40e_tx_prepare_vlan_flags - prepare generic TX VLAN tagging flags for HW * @skb: send buffer @@ -1276,27 +1303,6 @@ static int i40e_tx_prepare_vlan_flags(struct sk_buff *skb, } /** - * i40e_tx_csum - is checksum offload requested - * @tx_ring: ptr to the ring to send - * @skb: ptr to the skb we're sending - * @tx_flags: the collected send information - * @protocol: the send protocol - * - * Returns true if checksum offload is requested - **/ -static bool i40e_tx_csum(struct i40e_ring *tx_ring, struct sk_buff *skb, - u32 tx_flags, __be16 protocol) -{ - if ((skb->ip_summed != CHECKSUM_PARTIAL) && - !(tx_flags & I40E_TX_FLAGS_TXSW)) { - if (!(tx_flags & I40E_TX_FLAGS_HW_VLAN)) - return false; - } - - return skb->ip_summed == CHECKSUM_PARTIAL; -} - -/** * i40e_tso - set up the tso context descriptor * @tx_ring: ptr to the ring to send * @skb: ptr to the skb we're sending @@ -1482,15 +1488,16 @@ static void i40e_create_tx_ctx(struct i40e_ring *tx_ring, const u32 cd_tunneling, const u32 cd_l2tag2) { struct i40e_tx_context_desc *context_desc; + int i = tx_ring->next_to_use; if (!cd_type_cmd_tso_mss && !cd_tunneling && !cd_l2tag2) return; /* grab the next descriptor */ - context_desc = I40E_TX_CTXTDESC(tx_ring, tx_ring->next_to_use); - tx_ring->next_to_use++; - if (tx_ring->next_to_use == tx_ring->count) - tx_ring->next_to_use = 0; + context_desc = I40E_TX_CTXTDESC(tx_ring, i); + + i++; + tx_ring->next_to_use = (i < tx_ring->count) ? i : 0; /* cpu_to_le32 and assign to struct fields */ context_desc->tunneling_params = cpu_to_le32(cd_tunneling); @@ -1512,68 +1519,71 @@ static void i40e_tx_map(struct i40e_ring *tx_ring, struct sk_buff *skb, struct i40e_tx_buffer *first, u32 tx_flags, const u8 hdr_len, u32 td_cmd, u32 td_offset) { - struct skb_frag_struct *frag = &skb_shinfo(skb)->frags[0]; unsigned int data_len = skb->data_len; unsigned int size = skb_headlen(skb); - struct device *dev = tx_ring->dev; - u32 paylen = skb->len - hdr_len; - u16 i = tx_ring->next_to_use; + struct skb_frag_struct *frag; struct i40e_tx_buffer *tx_bi; struct i40e_tx_desc *tx_desc; - u32 buf_offset = 0; + u16 i = tx_ring->next_to_use; u32 td_tag = 0; dma_addr_t dma; u16 gso_segs; - dma = dma_map_single(dev, skb->data, size, DMA_TO_DEVICE); - if (dma_mapping_error(dev, dma)) - goto dma_error; - if (tx_flags & I40E_TX_FLAGS_HW_VLAN) { td_cmd |= I40E_TX_DESC_CMD_IL2TAG1; td_tag = (tx_flags & I40E_TX_FLAGS_VLAN_MASK) >> I40E_TX_FLAGS_VLAN_SHIFT; } + if (tx_flags & (I40E_TX_FLAGS_TSO | I40E_TX_FLAGS_FSO)) + gso_segs = skb_shinfo(skb)->gso_segs; + else + gso_segs = 1; + + /* multiply data chunks by size of headers */ + first->bytecount = skb->len - hdr_len + (gso_segs * hdr_len); + first->gso_segs = gso_segs; + first->skb = skb; + first->tx_flags = tx_flags; + + dma = dma_map_single(tx_ring->dev, skb->data, size, DMA_TO_DEVICE); + tx_desc = I40E_TX_DESC(tx_ring, i); - for (;;) { - while (size > I40E_MAX_DATA_PER_TXD) { - tx_desc->buffer_addr = cpu_to_le64(dma + buf_offset); + tx_bi = first; + + for (frag = &skb_shinfo(skb)->frags[0];; frag++) { + if (dma_mapping_error(tx_ring->dev, dma)) + goto dma_error; + + /* record length, and DMA address */ + dma_unmap_len_set(tx_bi, len, size); + dma_unmap_addr_set(tx_bi, dma, dma); + + tx_desc->buffer_addr = cpu_to_le64(dma); + + while (unlikely(size > I40E_MAX_DATA_PER_TXD)) { tx_desc->cmd_type_offset_bsz = build_ctob(td_cmd, td_offset, I40E_MAX_DATA_PER_TXD, td_tag); - buf_offset += I40E_MAX_DATA_PER_TXD; - size -= I40E_MAX_DATA_PER_TXD; - tx_desc++; i++; if (i == tx_ring->count) { tx_desc = I40E_TX_DESC(tx_ring, 0); i = 0; } - } - tx_bi = &tx_ring->tx_bi[i]; - tx_bi->length = buf_offset + size; - tx_bi->tx_flags = tx_flags; - tx_bi->dma = dma; + dma += I40E_MAX_DATA_PER_TXD; + size -= I40E_MAX_DATA_PER_TXD; - tx_desc->buffer_addr = cpu_to_le64(dma + buf_offset); - tx_desc->cmd_type_offset_bsz = build_ctob(td_cmd, td_offset, - size, td_tag); + tx_desc->buffer_addr = cpu_to_le64(dma); + } if (likely(!data_len)) break; - size = skb_frag_size(frag); - data_len -= size; - buf_offset = 0; - tx_flags |= I40E_TX_FLAGS_MAPPED_AS_PAGE; - - dma = skb_frag_dma_map(dev, frag, 0, size, DMA_TO_DEVICE); - if (dma_mapping_error(dev, dma)) - goto dma_error; + tx_desc->cmd_type_offset_bsz = build_ctob(td_cmd, td_offset, + size, td_tag); tx_desc++; i++; @@ -1582,31 +1592,25 @@ static void i40e_tx_map(struct i40e_ring *tx_ring, struct sk_buff *skb, i = 0; } - frag++; - } - - tx_desc->cmd_type_offset_bsz |= - cpu_to_le64((u64)I40E_TXD_CMD << I40E_TXD_QW1_CMD_SHIFT); + size = skb_frag_size(frag); + data_len -= size; - i++; - if (i == tx_ring->count) - i = 0; + dma = skb_frag_dma_map(tx_ring->dev, frag, 0, size, + DMA_TO_DEVICE); - tx_ring->next_to_use = i; + tx_bi = &tx_ring->tx_bi[i]; + } - if (tx_flags & (I40E_TX_FLAGS_TSO | I40E_TX_FLAGS_FSO)) - gso_segs = skb_shinfo(skb)->gso_segs; - else - gso_segs = 1; + tx_desc->cmd_type_offset_bsz = + build_ctob(td_cmd, td_offset, size, td_tag) | + cpu_to_le64((u64)I40E_TXD_CMD << I40E_TXD_QW1_CMD_SHIFT); - /* multiply data chunks by size of headers */ - tx_bi->bytecount = paylen + (gso_segs * hdr_len); - tx_bi->gso_segs = gso_segs; - tx_bi->skb = skb; + netdev_tx_sent_queue(netdev_get_tx_queue(tx_ring->netdev, + tx_ring->queue_index), + first->bytecount); - /* set the timestamp and next to watch values */ + /* set the timestamp */ first->time_stamp = jiffies; - first->next_to_watch = tx_desc; /* Force memory writes to complete before letting h/w * know there are new descriptors to fetch. (Only @@ -1615,16 +1619,27 @@ static void i40e_tx_map(struct i40e_ring *tx_ring, struct sk_buff *skb, */ wmb(); + /* set next_to_watch value indicating a packet is present */ + first->next_to_watch = tx_desc; + + i++; + if (i == tx_ring->count) + i = 0; + + tx_ring->next_to_use = i; + + /* notify HW of packet */ writel(i, tx_ring->tail); + return; dma_error: - dev_info(dev, "TX DMA map failed\n"); + dev_info(tx_ring->dev, "TX DMA map failed\n"); /* clear dma mappings for failed tx_bi map */ for (;;) { tx_bi = &tx_ring->tx_bi[i]; - i40e_unmap_tx_resource(tx_ring, tx_bi); + i40e_unmap_and_free_tx_resource(tx_ring, tx_bi); if (tx_bi == first) break; if (i == 0) @@ -1632,8 +1647,6 @@ dma_error: i--; } - dev_kfree_skb_any(skb); - tx_ring->next_to_use = i; } @@ -1758,16 +1771,16 @@ static netdev_tx_t i40e_xmit_frame_ring(struct sk_buff *skb, skb_tx_timestamp(skb); + /* always enable CRC insertion offload */ + td_cmd |= I40E_TX_DESC_CMD_ICRC; + /* Always offload the checksum, since it's in the data descriptor */ - if (i40e_tx_csum(tx_ring, skb, tx_flags, protocol)) + if (skb->ip_summed == CHECKSUM_PARTIAL) { tx_flags |= I40E_TX_FLAGS_CSUM; - /* always enable offload insertion */ - td_cmd |= I40E_TX_DESC_CMD_ICRC; - - if (tx_flags & I40E_TX_FLAGS_CSUM) i40e_tx_enable_csum(skb, tx_flags, &td_cmd, &td_offset, tx_ring, &cd_tunneling); + } i40e_create_tx_ctx(tx_ring, cd_type_cmd_tso_mss, cd_tunneling, cd_l2tag2); @@ -1801,7 +1814,7 @@ netdev_tx_t i40e_lan_xmit_frame(struct sk_buff *skb, struct net_device *netdev) { struct i40e_netdev_priv *np = netdev_priv(netdev); struct i40e_vsi *vsi = np->vsi; - struct i40e_ring *tx_ring = &vsi->tx_rings[skb->queue_mapping]; + struct i40e_ring *tx_ring = vsi->tx_rings[skb->queue_mapping]; /* hardware can't handle really short frames, hardware padding works * beyond this point diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.h b/drivers/net/ethernet/intel/i40e/i40e_txrx.h index b1d7722..db55d99 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_txrx.h +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.h @@ -102,23 +102,20 @@ #define I40E_TX_FLAGS_IPV6 (u32)(1 << 5) #define I40E_TX_FLAGS_FCCRC (u32)(1 << 6) #define I40E_TX_FLAGS_FSO (u32)(1 << 7) -#define I40E_TX_FLAGS_TXSW (u32)(1 << 8) -#define I40E_TX_FLAGS_MAPPED_AS_PAGE (u32)(1 << 9) #define I40E_TX_FLAGS_VLAN_MASK 0xffff0000 #define I40E_TX_FLAGS_VLAN_PRIO_MASK 0xe0000000 #define I40E_TX_FLAGS_VLAN_PRIO_SHIFT 29 #define I40E_TX_FLAGS_VLAN_SHIFT 16 struct i40e_tx_buffer { - struct sk_buff *skb; - dma_addr_t dma; - unsigned long time_stamp; - u16 length; - u32 tx_flags; struct i40e_tx_desc *next_to_watch; + unsigned long time_stamp; + struct sk_buff *skb; unsigned int bytecount; - u16 gso_segs; - u8 mapped_as_page; + unsigned short gso_segs; + DEFINE_DMA_UNMAP_ADDR(dma); + DEFINE_DMA_UNMAP_LEN(len); + u32 tx_flags; }; struct i40e_rx_buffer { @@ -129,18 +126,18 @@ struct i40e_rx_buffer { unsigned int page_offset; }; -struct i40e_tx_queue_stats { +struct i40e_queue_stats { u64 packets; u64 bytes; +}; + +struct i40e_tx_queue_stats { u64 restart_queue; u64 tx_busy; - u64 completed; u64 tx_done_old; }; struct i40e_rx_queue_stats { - u64 packets; - u64 bytes; u64 non_eop_descs; u64 alloc_rx_page_failed; u64 alloc_rx_buff_failed; @@ -183,6 +180,7 @@ enum i40e_ring_state_t { /* struct that defines a descriptor ring, associated with a VSI */ struct i40e_ring { + struct i40e_ring *next; /* pointer to next ring in q_vector */ void *desc; /* Descriptor ring memory */ struct device *dev; /* Used for DMA mapping */ struct net_device *netdev; /* netdev ring maps to */ @@ -219,6 +217,8 @@ struct i40e_ring { bool ring_active; /* is ring online or not */ /* stats structs */ + struct i40e_queue_stats stats; + struct u64_stats_sync syncp; union { struct i40e_tx_queue_stats tx_stats; struct i40e_rx_queue_stats rx_stats; @@ -229,6 +229,8 @@ struct i40e_ring { struct i40e_vsi *vsi; /* Backreference to associated VSI */ struct i40e_q_vector *q_vector; /* Backreference to associated vector */ + + struct rcu_head rcu; /* to avoid race on free */ } ____cacheline_internodealigned_in_smp; enum i40e_latency_range { @@ -238,9 +240,8 @@ enum i40e_latency_range { }; struct i40e_ring_container { -#define I40E_MAX_RINGPAIR_PER_VECTOR 8 /* array of pointers to rings */ - struct i40e_ring *ring[I40E_MAX_RINGPAIR_PER_VECTOR]; + struct i40e_ring *ring; unsigned int total_bytes; /* total bytes processed this int */ unsigned int total_packets; /* total packets processed this int */ u16 count; @@ -248,6 +249,10 @@ struct i40e_ring_container { u16 itr; }; +/* iterator for handling rings in ring container */ +#define i40e_for_each_ring(pos, head) \ + for (pos = (head).ring; pos != NULL; pos = pos->next) + void i40e_alloc_rx_buffers(struct i40e_ring *rxr, u16 cleaned_count); netdev_tx_t i40e_lan_xmit_frame(struct sk_buff *skb, struct net_device *netdev); void i40e_clean_tx_ring(struct i40e_ring *tx_ring); diff --git a/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c b/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c index 8967e58..0759698 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c +++ b/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c @@ -251,7 +251,7 @@ static void i40e_config_irq_link_list(struct i40e_vf *vf, u16 vsi_idx, reg_idx = I40E_VPINT_LNKLST0(vf->vf_id); else reg_idx = I40E_VPINT_LNKLSTN( - ((pf->hw.func_caps.num_msix_vectors_vf - 1) + (pf->hw.func_caps.num_msix_vectors_vf * vf->vf_id) + (vector_id - 1)); if (vecmap->rxq_map == 0 && vecmap->txq_map == 0) { @@ -383,7 +383,7 @@ static int i40e_config_vsi_tx_queue(struct i40e_vf *vf, u16 vsi_idx, /* associate this queue with the PCI VF function */ qtx_ctl = I40E_QTX_CTL_VF_QUEUE; - qtx_ctl |= ((hw->hmc.hmc_fn_id << I40E_QTX_CTL_PF_INDX_SHIFT) + qtx_ctl |= ((hw->pf_id << I40E_QTX_CTL_PF_INDX_SHIFT) & I40E_QTX_CTL_PF_INDX_MASK); qtx_ctl |= (((vf->vf_id + hw->func_caps.vf_base_id) << I40E_QTX_CTL_VFVM_INDX_SHIFT) diff --git a/drivers/net/ethernet/intel/igb/e1000_82575.h b/drivers/net/ethernet/intel/igb/e1000_82575.h index 74a1506..8c24377 100644 --- a/drivers/net/ethernet/intel/igb/e1000_82575.h +++ b/drivers/net/ethernet/intel/igb/e1000_82575.h @@ -28,14 +28,14 @@ #ifndef _E1000_82575_H_ #define _E1000_82575_H_ -extern void igb_shutdown_serdes_link_82575(struct e1000_hw *hw); -extern void igb_power_up_serdes_link_82575(struct e1000_hw *hw); -extern void igb_power_down_phy_copper_82575(struct e1000_hw *hw); -extern void igb_rx_fifo_flush_82575(struct e1000_hw *hw); -extern s32 igb_read_i2c_byte(struct e1000_hw *hw, u8 byte_offset, - u8 dev_addr, u8 *data); -extern s32 igb_write_i2c_byte(struct e1000_hw *hw, u8 byte_offset, - u8 dev_addr, u8 data); +void igb_shutdown_serdes_link_82575(struct e1000_hw *hw); +void igb_power_up_serdes_link_82575(struct e1000_hw *hw); +void igb_power_down_phy_copper_82575(struct e1000_hw *hw); +void igb_rx_fifo_flush_82575(struct e1000_hw *hw); +s32 igb_read_i2c_byte(struct e1000_hw *hw, u8 byte_offset, u8 dev_addr, + u8 *data); +s32 igb_write_i2c_byte(struct e1000_hw *hw, u8 byte_offset, u8 dev_addr, + u8 data); #define ID_LED_DEFAULT_82575_SERDES ((ID_LED_DEF1_DEF2 << 12) | \ (ID_LED_DEF1_DEF2 << 8) | \ diff --git a/drivers/net/ethernet/intel/igb/e1000_hw.h b/drivers/net/ethernet/intel/igb/e1000_hw.h index 37a9c06..2e166b2 100644 --- a/drivers/net/ethernet/intel/igb/e1000_hw.h +++ b/drivers/net/ethernet/intel/igb/e1000_hw.h @@ -562,11 +562,11 @@ struct e1000_hw { u8 revision_id; }; -extern struct net_device *igb_get_hw_dev(struct e1000_hw *hw); +struct net_device *igb_get_hw_dev(struct e1000_hw *hw); #define hw_dbg(format, arg...) \ netdev_dbg(igb_get_hw_dev(hw), format, ##arg) /* These functions must be implemented by drivers */ -s32 igb_read_pcie_cap_reg(struct e1000_hw *hw, u32 reg, u16 *value); -s32 igb_write_pcie_cap_reg(struct e1000_hw *hw, u32 reg, u16 *value); +s32 igb_read_pcie_cap_reg(struct e1000_hw *hw, u32 reg, u16 *value); +s32 igb_write_pcie_cap_reg(struct e1000_hw *hw, u32 reg, u16 *value); #endif /* _E1000_HW_H_ */ diff --git a/drivers/net/ethernet/intel/igb/e1000_i210.h b/drivers/net/ethernet/intel/igb/e1000_i210.h index dde3c4b..2d91371 100644 --- a/drivers/net/ethernet/intel/igb/e1000_i210.h +++ b/drivers/net/ethernet/intel/igb/e1000_i210.h @@ -28,26 +28,24 @@ #ifndef _E1000_I210_H_ #define _E1000_I210_H_ -extern s32 igb_update_flash_i210(struct e1000_hw *hw); -extern s32 igb_update_nvm_checksum_i210(struct e1000_hw *hw); -extern s32 igb_validate_nvm_checksum_i210(struct e1000_hw *hw); -extern s32 igb_write_nvm_srwr_i210(struct e1000_hw *hw, u16 offset, - u16 words, u16 *data); -extern s32 igb_read_nvm_srrd_i210(struct e1000_hw *hw, u16 offset, - u16 words, u16 *data); -extern s32 igb_acquire_swfw_sync_i210(struct e1000_hw *hw, u16 mask); -extern void igb_release_swfw_sync_i210(struct e1000_hw *hw, u16 mask); -extern s32 igb_acquire_nvm_i210(struct e1000_hw *hw); -extern void igb_release_nvm_i210(struct e1000_hw *hw); -extern s32 igb_valid_led_default_i210(struct e1000_hw *hw, u16 *data); -extern s32 igb_read_invm_version(struct e1000_hw *hw, - struct e1000_fw_version *invm_ver); -extern s32 igb_read_xmdio_reg(struct e1000_hw *hw, u16 addr, u8 dev_addr, - u16 *data); -extern s32 igb_write_xmdio_reg(struct e1000_hw *hw, u16 addr, u8 dev_addr, - u16 data); -extern s32 igb_init_nvm_params_i210(struct e1000_hw *hw); -extern bool igb_get_flash_presence_i210(struct e1000_hw *hw); +s32 igb_update_flash_i210(struct e1000_hw *hw); +s32 igb_update_nvm_checksum_i210(struct e1000_hw *hw); +s32 igb_validate_nvm_checksum_i210(struct e1000_hw *hw); +s32 igb_write_nvm_srwr_i210(struct e1000_hw *hw, u16 offset, u16 words, + u16 *data); +s32 igb_read_nvm_srrd_i210(struct e1000_hw *hw, u16 offset, u16 words, + u16 *data); +s32 igb_acquire_swfw_sync_i210(struct e1000_hw *hw, u16 mask); +void igb_release_swfw_sync_i210(struct e1000_hw *hw, u16 mask); +s32 igb_acquire_nvm_i210(struct e1000_hw *hw); +void igb_release_nvm_i210(struct e1000_hw *hw); +s32 igb_valid_led_default_i210(struct e1000_hw *hw, u16 *data); +s32 igb_read_invm_version(struct e1000_hw *hw, + struct e1000_fw_version *invm_ver); +s32 igb_read_xmdio_reg(struct e1000_hw *hw, u16 addr, u8 dev_addr, u16 *data); +s32 igb_write_xmdio_reg(struct e1000_hw *hw, u16 addr, u8 dev_addr, u16 data); +s32 igb_init_nvm_params_i210(struct e1000_hw *hw); +bool igb_get_flash_presence_i210(struct e1000_hw *hw); #define E1000_STM_OPCODE 0xDB00 #define E1000_EEPROM_FLASH_SIZE_WORD 0x11 diff --git a/drivers/net/ethernet/intel/igb/e1000_mac.h b/drivers/net/ethernet/intel/igb/e1000_mac.h index 5e13e83..e4cbe8e 100644 --- a/drivers/net/ethernet/intel/igb/e1000_mac.h +++ b/drivers/net/ethernet/intel/igb/e1000_mac.h @@ -86,6 +86,6 @@ enum e1000_mng_mode { #define E1000_MNG_DHCP_COOKIE_STATUS_VLAN 0x2 -extern void e1000_init_function_pointers_82575(struct e1000_hw *hw); +void e1000_init_function_pointers_82575(struct e1000_hw *hw); #endif diff --git a/drivers/net/ethernet/intel/igb/e1000_phy.c b/drivers/net/ethernet/intel/igb/e1000_phy.c index e726675..c4c4fe3 100644 --- a/drivers/net/ethernet/intel/igb/e1000_phy.c +++ b/drivers/net/ethernet/intel/igb/e1000_phy.c @@ -708,11 +708,6 @@ s32 igb_copper_link_setup_m88(struct e1000_hw *hw) hw_dbg("Error committing the PHY changes\n"); goto out; } - if (phy->type == e1000_phy_i210) { - ret_val = igb_set_master_slave_mode(hw); - if (ret_val) - return ret_val; - } out: return ret_val; @@ -806,6 +801,9 @@ s32 igb_copper_link_setup_m88_gen2(struct e1000_hw *hw) hw_dbg("Error committing the PHY changes\n"); return ret_val; } + ret_val = igb_set_master_slave_mode(hw); + if (ret_val) + return ret_val; return 0; } diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h index 6807b09..5e9ed89 100644 --- a/drivers/net/ethernet/intel/igb/igb.h +++ b/drivers/net/ethernet/intel/igb/igb.h @@ -483,40 +483,38 @@ enum igb_boards { extern char igb_driver_name[]; extern char igb_driver_version[]; -extern int igb_up(struct igb_adapter *); -extern void igb_down(struct igb_adapter *); -extern void igb_reinit_locked(struct igb_adapter *); -extern void igb_reset(struct igb_adapter *); -extern void igb_write_rss_indir_tbl(struct igb_adapter *); -extern int igb_set_spd_dplx(struct igb_adapter *, u32, u8); -extern int igb_setup_tx_resources(struct igb_ring *); -extern int igb_setup_rx_resources(struct igb_ring *); -extern void igb_free_tx_resources(struct igb_ring *); -extern void igb_free_rx_resources(struct igb_ring *); -extern void igb_configure_tx_ring(struct igb_adapter *, struct igb_ring *); -extern void igb_configure_rx_ring(struct igb_adapter *, struct igb_ring *); -extern void igb_setup_tctl(struct igb_adapter *); -extern void igb_setup_rctl(struct igb_adapter *); -extern netdev_tx_t igb_xmit_frame_ring(struct sk_buff *, struct igb_ring *); -extern void igb_unmap_and_free_tx_resource(struct igb_ring *, - struct igb_tx_buffer *); -extern void igb_alloc_rx_buffers(struct igb_ring *, u16); -extern void igb_update_stats(struct igb_adapter *, struct rtnl_link_stats64 *); -extern bool igb_has_link(struct igb_adapter *adapter); -extern void igb_set_ethtool_ops(struct net_device *); -extern void igb_power_up_link(struct igb_adapter *); -extern void igb_set_fw_version(struct igb_adapter *); -extern void igb_ptp_init(struct igb_adapter *adapter); -extern void igb_ptp_stop(struct igb_adapter *adapter); -extern void igb_ptp_reset(struct igb_adapter *adapter); -extern void igb_ptp_tx_work(struct work_struct *work); -extern void igb_ptp_rx_hang(struct igb_adapter *adapter); -extern void igb_ptp_tx_hwtstamp(struct igb_adapter *adapter); -extern void igb_ptp_rx_rgtstamp(struct igb_q_vector *q_vector, - struct sk_buff *skb); -extern void igb_ptp_rx_pktstamp(struct igb_q_vector *q_vector, - unsigned char *va, - struct sk_buff *skb); +int igb_up(struct igb_adapter *); +void igb_down(struct igb_adapter *); +void igb_reinit_locked(struct igb_adapter *); +void igb_reset(struct igb_adapter *); +int igb_reinit_queues(struct igb_adapter *); +void igb_write_rss_indir_tbl(struct igb_adapter *); +int igb_set_spd_dplx(struct igb_adapter *, u32, u8); +int igb_setup_tx_resources(struct igb_ring *); +int igb_setup_rx_resources(struct igb_ring *); +void igb_free_tx_resources(struct igb_ring *); +void igb_free_rx_resources(struct igb_ring *); +void igb_configure_tx_ring(struct igb_adapter *, struct igb_ring *); +void igb_configure_rx_ring(struct igb_adapter *, struct igb_ring *); +void igb_setup_tctl(struct igb_adapter *); +void igb_setup_rctl(struct igb_adapter *); +netdev_tx_t igb_xmit_frame_ring(struct sk_buff *, struct igb_ring *); +void igb_unmap_and_free_tx_resource(struct igb_ring *, struct igb_tx_buffer *); +void igb_alloc_rx_buffers(struct igb_ring *, u16); +void igb_update_stats(struct igb_adapter *, struct rtnl_link_stats64 *); +bool igb_has_link(struct igb_adapter *adapter); +void igb_set_ethtool_ops(struct net_device *); +void igb_power_up_link(struct igb_adapter *); +void igb_set_fw_version(struct igb_adapter *); +void igb_ptp_init(struct igb_adapter *adapter); +void igb_ptp_stop(struct igb_adapter *adapter); +void igb_ptp_reset(struct igb_adapter *adapter); +void igb_ptp_tx_work(struct work_struct *work); +void igb_ptp_rx_hang(struct igb_adapter *adapter); +void igb_ptp_tx_hwtstamp(struct igb_adapter *adapter); +void igb_ptp_rx_rgtstamp(struct igb_q_vector *q_vector, struct sk_buff *skb); +void igb_ptp_rx_pktstamp(struct igb_q_vector *q_vector, unsigned char *va, + struct sk_buff *skb); static inline void igb_ptp_rx_hwtstamp(struct igb_ring *rx_ring, union e1000_adv_rx_desc *rx_desc, struct sk_buff *skb) @@ -531,11 +529,11 @@ static inline void igb_ptp_rx_hwtstamp(struct igb_ring *rx_ring, rx_ring->last_rx_timestamp = jiffies; } -extern int igb_ptp_hwtstamp_ioctl(struct net_device *netdev, - struct ifreq *ifr, int cmd); +int igb_ptp_hwtstamp_ioctl(struct net_device *netdev, struct ifreq *ifr, + int cmd); #ifdef CONFIG_IGB_HWMON -extern void igb_sysfs_exit(struct igb_adapter *adapter); -extern int igb_sysfs_init(struct igb_adapter *adapter); +void igb_sysfs_exit(struct igb_adapter *adapter); +int igb_sysfs_init(struct igb_adapter *adapter); #endif static inline s32 igb_reset_phy(struct e1000_hw *hw) { diff --git a/drivers/net/ethernet/intel/igb/igb_ethtool.c b/drivers/net/ethernet/intel/igb/igb_ethtool.c index 151e00c..b0f3666 100644 --- a/drivers/net/ethernet/intel/igb/igb_ethtool.c +++ b/drivers/net/ethernet/intel/igb/igb_ethtool.c @@ -146,6 +146,7 @@ static int igb_get_settings(struct net_device *netdev, struct ethtool_cmd *ecmd) struct e1000_sfp_flags *eth_flags = &dev_spec->eth_flags; u32 status; + status = rd32(E1000_STATUS); if (hw->phy.media_type == e1000_media_type_copper) { ecmd->supported = (SUPPORTED_10baseT_Half | @@ -169,13 +170,22 @@ static int igb_get_settings(struct net_device *netdev, struct ethtool_cmd *ecmd) ecmd->transceiver = XCVR_INTERNAL; } else { ecmd->supported = (SUPPORTED_FIBRE | + SUPPORTED_1000baseKX_Full | SUPPORTED_Autoneg | SUPPORTED_Pause); - ecmd->advertising = ADVERTISED_FIBRE; - - if ((eth_flags->e1000_base_lx) || (eth_flags->e1000_base_sx)) { - ecmd->supported |= SUPPORTED_1000baseT_Full; - ecmd->advertising |= ADVERTISED_1000baseT_Full; + ecmd->advertising = (ADVERTISED_FIBRE | + ADVERTISED_1000baseKX_Full); + if (hw->mac.type == e1000_i354) { + if ((hw->device_id == + E1000_DEV_ID_I354_BACKPLANE_2_5GBPS) && + !(status & E1000_STATUS_2P5_SKU_OVER)) { + ecmd->supported |= SUPPORTED_2500baseX_Full; + ecmd->supported &= + ~SUPPORTED_1000baseKX_Full; + ecmd->advertising |= ADVERTISED_2500baseX_Full; + ecmd->advertising &= + ~ADVERTISED_1000baseKX_Full; + } } if (eth_flags->e100_base_fx) { ecmd->supported |= SUPPORTED_100baseT_Full; @@ -187,35 +197,29 @@ static int igb_get_settings(struct net_device *netdev, struct ethtool_cmd *ecmd) ecmd->port = PORT_FIBRE; ecmd->transceiver = XCVR_EXTERNAL; } - if (hw->mac.autoneg != 1) ecmd->advertising &= ~(ADVERTISED_Pause | ADVERTISED_Asym_Pause); - if (hw->fc.requested_mode == e1000_fc_full) + switch (hw->fc.requested_mode) { + case e1000_fc_full: ecmd->advertising |= ADVERTISED_Pause; - else if (hw->fc.requested_mode == e1000_fc_rx_pause) + break; + case e1000_fc_rx_pause: ecmd->advertising |= (ADVERTISED_Pause | ADVERTISED_Asym_Pause); - else if (hw->fc.requested_mode == e1000_fc_tx_pause) + break; + case e1000_fc_tx_pause: ecmd->advertising |= ADVERTISED_Asym_Pause; - else + break; + default: ecmd->advertising &= ~(ADVERTISED_Pause | ADVERTISED_Asym_Pause); - - status = rd32(E1000_STATUS); - + } if (status & E1000_STATUS_LU) { - if (hw->mac.type == e1000_i354) { - if ((status & E1000_STATUS_2P5_SKU) && - !(status & E1000_STATUS_2P5_SKU_OVER)) { - ecmd->supported = SUPPORTED_2500baseX_Full; - ecmd->advertising = ADVERTISED_2500baseX_Full; - ecmd->speed = SPEED_2500; - } else { - ecmd->supported = SUPPORTED_1000baseT_Full; - ecmd->advertising = ADVERTISED_1000baseT_Full; - } + if ((status & E1000_STATUS_2P5_SKU) && + !(status & E1000_STATUS_2P5_SKU_OVER)) { + ecmd->speed = SPEED_2500; } else if (status & E1000_STATUS_SPEED_1000) { ecmd->speed = SPEED_1000; } else if (status & E1000_STATUS_SPEED_100) { @@ -232,7 +236,6 @@ static int igb_get_settings(struct net_device *netdev, struct ethtool_cmd *ecmd) ecmd->speed = -1; ecmd->duplex = -1; } - if ((hw->phy.media_type == e1000_media_type_fiber) || hw->mac.autoneg) ecmd->autoneg = AUTONEG_ENABLE; @@ -771,8 +774,10 @@ static int igb_set_eeprom(struct net_device *netdev, if (eeprom->len == 0) return -EOPNOTSUPP; - if (hw->mac.type == e1000_i211) + if ((hw->mac.type >= e1000_i210) && + !igb_get_flash_presence_i210(hw)) { return -EOPNOTSUPP; + } if (eeprom->magic != (hw->vendor_id | (hw->device_id << 16))) return -EFAULT; @@ -1659,7 +1664,8 @@ static int igb_setup_loopback_test(struct igb_adapter *adapter) if ((hw->device_id == E1000_DEV_ID_DH89XXCC_SGMII) || (hw->device_id == E1000_DEV_ID_DH89XXCC_SERDES) || (hw->device_id == E1000_DEV_ID_DH89XXCC_BACKPLANE) || - (hw->device_id == E1000_DEV_ID_DH89XXCC_SFP)) { + (hw->device_id == E1000_DEV_ID_DH89XXCC_SFP) || + (hw->device_id == E1000_DEV_ID_I354_SGMII)) { /* Enable DH89xxCC MPHY for near end loopback */ reg = rd32(E1000_MPHY_ADDR_CTL); @@ -1725,7 +1731,8 @@ static void igb_loopback_cleanup(struct igb_adapter *adapter) if ((hw->device_id == E1000_DEV_ID_DH89XXCC_SGMII) || (hw->device_id == E1000_DEV_ID_DH89XXCC_SERDES) || (hw->device_id == E1000_DEV_ID_DH89XXCC_BACKPLANE) || - (hw->device_id == E1000_DEV_ID_DH89XXCC_SFP)) { + (hw->device_id == E1000_DEV_ID_DH89XXCC_SFP) || + (hw->device_id == E1000_DEV_ID_I354_SGMII)) { u32 reg; /* Disable near end loopback on DH89xxCC */ @@ -2877,6 +2884,88 @@ static int igb_set_rxfh_indir(struct net_device *netdev, const u32 *indir) return 0; } +static unsigned int igb_max_channels(struct igb_adapter *adapter) +{ + struct e1000_hw *hw = &adapter->hw; + unsigned int max_combined = 0; + + switch (hw->mac.type) { + case e1000_i211: + max_combined = IGB_MAX_RX_QUEUES_I211; + break; + case e1000_82575: + case e1000_i210: + max_combined = IGB_MAX_RX_QUEUES_82575; + break; + case e1000_i350: + if (!!adapter->vfs_allocated_count) { + max_combined = 1; + break; + } + /* fall through */ + case e1000_82576: + if (!!adapter->vfs_allocated_count) { + max_combined = 2; + break; + } + /* fall through */ + case e1000_82580: + case e1000_i354: + default: + max_combined = IGB_MAX_RX_QUEUES; + break; + } + + return max_combined; +} + +static void igb_get_channels(struct net_device *netdev, + struct ethtool_channels *ch) +{ + struct igb_adapter *adapter = netdev_priv(netdev); + + /* Report maximum channels */ + ch->max_combined = igb_max_channels(adapter); + + /* Report info for other vector */ + if (adapter->msix_entries) { + ch->max_other = NON_Q_VECTORS; + ch->other_count = NON_Q_VECTORS; + } + + ch->combined_count = adapter->rss_queues; +} + +static int igb_set_channels(struct net_device *netdev, + struct ethtool_channels *ch) +{ + struct igb_adapter *adapter = netdev_priv(netdev); + unsigned int count = ch->combined_count; + + /* Verify they are not requesting separate vectors */ + if (!count || ch->rx_count || ch->tx_count) + return -EINVAL; + + /* Verify other_count is valid and has not been changed */ + if (ch->other_count != NON_Q_VECTORS) + return -EINVAL; + + /* Verify the number of channels doesn't exceed hw limits */ + if (count > igb_max_channels(adapter)) + return -EINVAL; + + if (count != adapter->rss_queues) { + adapter->rss_queues = count; + + /* Hardware has to reinitialize queues and interrupts to + * match the new configuration. + */ + return igb_reinit_queues(adapter); + } + + return 0; +} + static const struct ethtool_ops igb_ethtool_ops = { .get_settings = igb_get_settings, .set_settings = igb_set_settings, @@ -2913,6 +3002,8 @@ static const struct ethtool_ops igb_ethtool_ops = { .get_rxfh_indir_size = igb_get_rxfh_indir_size, .get_rxfh_indir = igb_get_rxfh_indir, .set_rxfh_indir = igb_set_rxfh_indir, + .get_channels = igb_get_channels, + .set_channels = igb_set_channels, .begin = igb_ethtool_begin, .complete = igb_ethtool_complete, }; diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index 8cf44f2..ebe6370 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -182,6 +182,7 @@ static void igb_check_vf_rate_limit(struct igb_adapter *); #ifdef CONFIG_PCI_IOV static int igb_vf_configure(struct igb_adapter *adapter, int vf); +static int igb_pci_enable_sriov(struct pci_dev *dev, int num_vfs); #endif #ifdef CONFIG_PM @@ -2429,7 +2430,7 @@ err_dma: } #ifdef CONFIG_PCI_IOV -static int igb_disable_sriov(struct pci_dev *pdev) +static int igb_disable_sriov(struct pci_dev *pdev) { struct net_device *netdev = pci_get_drvdata(pdev); struct igb_adapter *adapter = netdev_priv(netdev); @@ -2470,27 +2471,19 @@ static int igb_enable_sriov(struct pci_dev *pdev, int num_vfs) int err = 0; int i; - if (!adapter->msix_entries) { + if (!adapter->msix_entries || num_vfs > 7) { err = -EPERM; goto out; } - if (!num_vfs) goto out; - else if (old_vfs && old_vfs == num_vfs) - goto out; - else if (old_vfs && old_vfs != num_vfs) - err = igb_disable_sriov(pdev); - - if (err) - goto out; - - if (num_vfs > 7) { - err = -EPERM; - goto out; - } - adapter->vfs_allocated_count = num_vfs; + if (old_vfs) { + dev_info(&pdev->dev, "%d pre-allocated VFs found - override max_vfs setting of %d\n", + old_vfs, max_vfs); + adapter->vfs_allocated_count = old_vfs; + } else + adapter->vfs_allocated_count = num_vfs; adapter->vf_data = kcalloc(adapter->vfs_allocated_count, sizeof(struct vf_data_storage), GFP_KERNEL); @@ -2504,10 +2497,12 @@ static int igb_enable_sriov(struct pci_dev *pdev, int num_vfs) goto out; } - err = pci_enable_sriov(pdev, adapter->vfs_allocated_count); - if (err) - goto err_out; - + /* only call pci_enable_sriov() if no VFs are allocated already */ + if (!old_vfs) { + err = pci_enable_sriov(pdev, adapter->vfs_allocated_count); + if (err) + goto err_out; + } dev_info(&pdev->dev, "%d VFs allocated\n", adapter->vfs_allocated_count); for (i = 0; i < adapter->vfs_allocated_count; i++) @@ -2623,7 +2618,7 @@ static void igb_probe_vfs(struct igb_adapter *adapter) return; pci_sriov_set_totalvfs(pdev, 7); - igb_enable_sriov(pdev, max_vfs); + igb_pci_enable_sriov(pdev, max_vfs); #endif /* CONFIG_PCI_IOV */ } @@ -5708,7 +5703,7 @@ static void igb_vf_reset_msg(struct igb_adapter *adapter, u32 vf) /* reply to reset with ack and vf mac address */ msgbuf[0] = E1000_VF_RESET | E1000_VT_MSGTYPE_ACK; - memcpy(addr, vf_mac, 6); + memcpy(addr, vf_mac, ETH_ALEN); igb_write_mbx(hw, msgbuf, 3, vf); } @@ -7838,4 +7833,26 @@ s32 igb_write_i2c_byte(struct e1000_hw *hw, u8 byte_offset, return E1000_SUCCESS; } + +int igb_reinit_queues(struct igb_adapter *adapter) +{ + struct net_device *netdev = adapter->netdev; + struct pci_dev *pdev = adapter->pdev; + int err = 0; + + if (netif_running(netdev)) + igb_close(netdev); + + igb_clear_interrupt_scheme(adapter); + + if (igb_init_interrupt_scheme(adapter, true)) { + dev_err(&pdev->dev, "Unable to allocate memory for queues\n"); + return -ENOMEM; + } + + if (netif_running(netdev)) + err = igb_open(netdev); + + return err; +} /* igb_main.c */ diff --git a/drivers/net/ethernet/intel/igbvf/igbvf.h b/drivers/net/ethernet/intel/igbvf/igbvf.h index a1463e3..7d6a25c 100644 --- a/drivers/net/ethernet/intel/igbvf/igbvf.h +++ b/drivers/net/ethernet/intel/igbvf/igbvf.h @@ -312,17 +312,17 @@ enum igbvf_state_t { extern char igbvf_driver_name[]; extern const char igbvf_driver_version[]; -extern void igbvf_check_options(struct igbvf_adapter *); -extern void igbvf_set_ethtool_ops(struct net_device *); - -extern int igbvf_up(struct igbvf_adapter *); -extern void igbvf_down(struct igbvf_adapter *); -extern void igbvf_reinit_locked(struct igbvf_adapter *); -extern int igbvf_setup_rx_resources(struct igbvf_adapter *, struct igbvf_ring *); -extern int igbvf_setup_tx_resources(struct igbvf_adapter *, struct igbvf_ring *); -extern void igbvf_free_rx_resources(struct igbvf_ring *); -extern void igbvf_free_tx_resources(struct igbvf_ring *); -extern void igbvf_update_stats(struct igbvf_adapter *); +void igbvf_check_options(struct igbvf_adapter *); +void igbvf_set_ethtool_ops(struct net_device *); + +int igbvf_up(struct igbvf_adapter *); +void igbvf_down(struct igbvf_adapter *); +void igbvf_reinit_locked(struct igbvf_adapter *); +int igbvf_setup_rx_resources(struct igbvf_adapter *, struct igbvf_ring *); +int igbvf_setup_tx_resources(struct igbvf_adapter *, struct igbvf_ring *); +void igbvf_free_rx_resources(struct igbvf_ring *); +void igbvf_free_tx_resources(struct igbvf_ring *); +void igbvf_update_stats(struct igbvf_adapter *); extern unsigned int copybreak; diff --git a/drivers/net/ethernet/intel/igbvf/netdev.c b/drivers/net/ethernet/intel/igbvf/netdev.c index 93eb7ee..9fadbb2 100644 --- a/drivers/net/ethernet/intel/igbvf/netdev.c +++ b/drivers/net/ethernet/intel/igbvf/netdev.c @@ -2343,10 +2343,9 @@ static int igbvf_change_mtu(struct net_device *netdev, int new_mtu) struct igbvf_adapter *adapter = netdev_priv(netdev); int max_frame = new_mtu + ETH_HLEN + ETH_FCS_LEN; - if ((new_mtu < 68) || (max_frame > MAX_JUMBO_FRAME_SIZE)) { - dev_err(&adapter->pdev->dev, "Invalid MTU setting\n"); + if (new_mtu < 68 || new_mtu > INT_MAX - ETH_HLEN - ETH_FCS_LEN || + max_frame > MAX_JUMBO_FRAME_SIZE) return -EINVAL; - } #define MAX_STD_JUMBO_FRAME_SIZE 9234 if (max_frame > MAX_STD_JUMBO_FRAME_SIZE) { @@ -2699,7 +2698,7 @@ static int igbvf_probe(struct pci_dev *pdev, const struct pci_device_id *ent) if (ei->get_variants) { err = ei->get_variants(adapter); if (err) - goto err_ioremap; + goto err_get_variants; } /* setup adapter struct */ @@ -2796,6 +2795,7 @@ err_hw_init: kfree(adapter->rx_ring); err_sw_init: igbvf_reset_interrupt_capability(adapter); +err_get_variants: iounmap(adapter->hw.hw_addr); err_ioremap: free_netdev(netdev); diff --git a/drivers/net/ethernet/intel/igbvf/vf.c b/drivers/net/ethernet/intel/igbvf/vf.c index eea0e10..955ad8c 100644 --- a/drivers/net/ethernet/intel/igbvf/vf.c +++ b/drivers/net/ethernet/intel/igbvf/vf.c @@ -154,7 +154,7 @@ static s32 e1000_reset_hw_vf(struct e1000_hw *hw) ret_val = mbx->ops.read_posted(hw, msgbuf, 3); if (!ret_val) { if (msgbuf[0] == (E1000_VF_RESET | E1000_VT_MSGTYPE_ACK)) - memcpy(hw->mac.perm_addr, addr, 6); + memcpy(hw->mac.perm_addr, addr, ETH_ALEN); else ret_val = -E1000_ERR_MAC_INIT; } @@ -314,7 +314,7 @@ static void e1000_rar_set_vf(struct e1000_hw *hw, u8 * addr, u32 index) memset(msgbuf, 0, 12); msgbuf[0] = E1000_VF_SET_MAC_ADDR; - memcpy(msg_addr, addr, 6); + memcpy(msg_addr, addr, ETH_ALEN); ret_val = mbx->ops.write_posted(hw, msgbuf, 3); if (!ret_val) diff --git a/drivers/net/ethernet/intel/ixgb/ixgb.h b/drivers/net/ethernet/intel/ixgb/ixgb.h index 4d2ae97..2224cc2 100644 --- a/drivers/net/ethernet/intel/ixgb/ixgb.h +++ b/drivers/net/ethernet/intel/ixgb/ixgb.h @@ -187,21 +187,21 @@ enum ixgb_state_t { }; /* Exported from other modules */ -extern void ixgb_check_options(struct ixgb_adapter *adapter); -extern void ixgb_set_ethtool_ops(struct net_device *netdev); +void ixgb_check_options(struct ixgb_adapter *adapter); +void ixgb_set_ethtool_ops(struct net_device *netdev); extern char ixgb_driver_name[]; extern const char ixgb_driver_version[]; -extern void ixgb_set_speed_duplex(struct net_device *netdev); +void ixgb_set_speed_duplex(struct net_device *netdev); -extern int ixgb_up(struct ixgb_adapter *adapter); -extern void ixgb_down(struct ixgb_adapter *adapter, bool kill_watchdog); -extern void ixgb_reset(struct ixgb_adapter *adapter); -extern int ixgb_setup_rx_resources(struct ixgb_adapter *adapter); -extern int ixgb_setup_tx_resources(struct ixgb_adapter *adapter); -extern void ixgb_free_rx_resources(struct ixgb_adapter *adapter); -extern void ixgb_free_tx_resources(struct ixgb_adapter *adapter); -extern void ixgb_update_stats(struct ixgb_adapter *adapter); +int ixgb_up(struct ixgb_adapter *adapter); +void ixgb_down(struct ixgb_adapter *adapter, bool kill_watchdog); +void ixgb_reset(struct ixgb_adapter *adapter); +int ixgb_setup_rx_resources(struct ixgb_adapter *adapter); +int ixgb_setup_tx_resources(struct ixgb_adapter *adapter); +void ixgb_free_rx_resources(struct ixgb_adapter *adapter); +void ixgb_free_tx_resources(struct ixgb_adapter *adapter); +void ixgb_update_stats(struct ixgb_adapter *adapter); #endif /* _IXGB_H_ */ diff --git a/drivers/net/ethernet/intel/ixgb/ixgb_hw.h b/drivers/net/ethernet/intel/ixgb/ixgb_hw.h index 2a99a35..0bd5d72 100644 --- a/drivers/net/ethernet/intel/ixgb/ixgb_hw.h +++ b/drivers/net/ethernet/intel/ixgb/ixgb_hw.h @@ -759,27 +759,20 @@ struct ixgb_hw_stats { }; /* Function Prototypes */ -extern bool ixgb_adapter_stop(struct ixgb_hw *hw); -extern bool ixgb_init_hw(struct ixgb_hw *hw); -extern bool ixgb_adapter_start(struct ixgb_hw *hw); -extern void ixgb_check_for_link(struct ixgb_hw *hw); -extern bool ixgb_check_for_bad_link(struct ixgb_hw *hw); - -extern void ixgb_rar_set(struct ixgb_hw *hw, - u8 *addr, - u32 index); +bool ixgb_adapter_stop(struct ixgb_hw *hw); +bool ixgb_init_hw(struct ixgb_hw *hw); +bool ixgb_adapter_start(struct ixgb_hw *hw); +void ixgb_check_for_link(struct ixgb_hw *hw); +bool ixgb_check_for_bad_link(struct ixgb_hw *hw); +void ixgb_rar_set(struct ixgb_hw *hw, u8 *addr, u32 index); /* Filters (multicast, vlan, receive) */ -extern void ixgb_mc_addr_list_update(struct ixgb_hw *hw, - u8 *mc_addr_list, - u32 mc_addr_count, - u32 pad); +void ixgb_mc_addr_list_update(struct ixgb_hw *hw, u8 *mc_addr_list, + u32 mc_addr_count, u32 pad); /* Vfta functions */ -extern void ixgb_write_vfta(struct ixgb_hw *hw, - u32 offset, - u32 value); +void ixgb_write_vfta(struct ixgb_hw *hw, u32 offset, u32 value); /* Access functions to eeprom data */ void ixgb_get_ee_mac_addr(struct ixgb_hw *hw, u8 *mac_addr); diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe.h b/drivers/net/ethernet/intel/ixgbe/ixgbe.h index 0ac6b11..f38fc0a 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe.h +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe.h @@ -55,7 +55,7 @@ #include <net/busy_poll.h> #ifdef CONFIG_NET_RX_BUSY_POLL -#define LL_EXTENDED_STATS +#define BP_EXTENDED_STATS #endif /* common prefix used by pr_<> macros */ #undef pr_fmt @@ -67,7 +67,11 @@ #define IXGBE_MAX_TXD 4096 #define IXGBE_MIN_TXD 64 +#if (PAGE_SIZE < 8192) #define IXGBE_DEFAULT_RXD 512 +#else +#define IXGBE_DEFAULT_RXD 128 +#endif #define IXGBE_MAX_RXD 4096 #define IXGBE_MIN_RXD 64 @@ -187,11 +191,11 @@ struct ixgbe_rx_buffer { struct ixgbe_queue_stats { u64 packets; u64 bytes; -#ifdef LL_EXTENDED_STATS +#ifdef BP_EXTENDED_STATS u64 yields; u64 misses; u64 cleaned; -#endif /* LL_EXTENDED_STATS */ +#endif /* BP_EXTENDED_STATS */ }; struct ixgbe_tx_queue_stats { @@ -219,6 +223,15 @@ enum ixgbe_ring_state_t { __IXGBE_RX_FCOE, }; +struct ixgbe_fwd_adapter { + unsigned long active_vlans[BITS_TO_LONGS(VLAN_N_VID)]; + struct net_device *netdev; + struct ixgbe_adapter *real_adapter; + unsigned int tx_base_queue; + unsigned int rx_base_queue; + int pool; +}; + #define check_for_tx_hang(ring) \ test_bit(__IXGBE_TX_DETECT_HANG, &(ring)->state) #define set_check_for_tx_hang(ring) \ @@ -236,6 +249,7 @@ struct ixgbe_ring { struct ixgbe_q_vector *q_vector; /* backpointer to host q_vector */ struct net_device *netdev; /* netdev ring belongs to */ struct device *dev; /* device for DMA mapping */ + struct ixgbe_fwd_adapter *l2_accel_priv; void *desc; /* descriptor ring memory */ union { struct ixgbe_tx_buffer *tx_buffer_info; @@ -293,6 +307,12 @@ enum ixgbe_ring_f_enum { #define IXGBE_MAX_FCOE_INDICES 8 #define MAX_RX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1) #define MAX_TX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1) +#define IXGBE_MAX_L2A_QUEUES 4 +#define IXGBE_MAX_L2A_QUEUES 4 +#define IXGBE_BAD_L2A_QUEUE 3 +#define IXGBE_MAX_MACVLANS 31 +#define IXGBE_MAX_DCBMACVLANS 8 + struct ixgbe_ring_feature { u16 limit; /* upper limit on feature indices */ u16 indices; /* current value of indices */ @@ -369,11 +389,13 @@ struct ixgbe_q_vector { #ifdef CONFIG_NET_RX_BUSY_POLL unsigned int state; #define IXGBE_QV_STATE_IDLE 0 -#define IXGBE_QV_STATE_NAPI 1 /* NAPI owns this QV */ -#define IXGBE_QV_STATE_POLL 2 /* poll owns this QV */ -#define IXGBE_QV_LOCKED (IXGBE_QV_STATE_NAPI | IXGBE_QV_STATE_POLL) -#define IXGBE_QV_STATE_NAPI_YIELD 4 /* NAPI yielded this QV */ -#define IXGBE_QV_STATE_POLL_YIELD 8 /* poll yielded this QV */ +#define IXGBE_QV_STATE_NAPI 1 /* NAPI owns this QV */ +#define IXGBE_QV_STATE_POLL 2 /* poll owns this QV */ +#define IXGBE_QV_STATE_DISABLED 4 /* QV is disabled */ +#define IXGBE_QV_OWNED (IXGBE_QV_STATE_NAPI | IXGBE_QV_STATE_POLL) +#define IXGBE_QV_LOCKED (IXGBE_QV_OWNED | IXGBE_QV_STATE_DISABLED) +#define IXGBE_QV_STATE_NAPI_YIELD 8 /* NAPI yielded this QV */ +#define IXGBE_QV_STATE_POLL_YIELD 16 /* poll yielded this QV */ #define IXGBE_QV_YIELD (IXGBE_QV_STATE_NAPI_YIELD | IXGBE_QV_STATE_POLL_YIELD) #define IXGBE_QV_USER_PEND (IXGBE_QV_STATE_POLL | IXGBE_QV_STATE_POLL_YIELD) spinlock_t lock; @@ -394,18 +416,18 @@ static inline void ixgbe_qv_init_lock(struct ixgbe_q_vector *q_vector) static inline bool ixgbe_qv_lock_napi(struct ixgbe_q_vector *q_vector) { int rc = true; - spin_lock(&q_vector->lock); + spin_lock_bh(&q_vector->lock); if (q_vector->state & IXGBE_QV_LOCKED) { WARN_ON(q_vector->state & IXGBE_QV_STATE_NAPI); q_vector->state |= IXGBE_QV_STATE_NAPI_YIELD; rc = false; -#ifdef LL_EXTENDED_STATS +#ifdef BP_EXTENDED_STATS q_vector->tx.ring->stats.yields++; #endif } else /* we don't care if someone yielded */ q_vector->state = IXGBE_QV_STATE_NAPI; - spin_unlock(&q_vector->lock); + spin_unlock_bh(&q_vector->lock); return rc; } @@ -413,14 +435,15 @@ static inline bool ixgbe_qv_lock_napi(struct ixgbe_q_vector *q_vector) static inline bool ixgbe_qv_unlock_napi(struct ixgbe_q_vector *q_vector) { int rc = false; - spin_lock(&q_vector->lock); + spin_lock_bh(&q_vector->lock); WARN_ON(q_vector->state & (IXGBE_QV_STATE_POLL | IXGBE_QV_STATE_NAPI_YIELD)); if (q_vector->state & IXGBE_QV_STATE_POLL_YIELD) rc = true; - q_vector->state = IXGBE_QV_STATE_IDLE; - spin_unlock(&q_vector->lock); + /* will reset state to idle, unless QV is disabled */ + q_vector->state &= IXGBE_QV_STATE_DISABLED; + spin_unlock_bh(&q_vector->lock); return rc; } @@ -432,7 +455,7 @@ static inline bool ixgbe_qv_lock_poll(struct ixgbe_q_vector *q_vector) if ((q_vector->state & IXGBE_QV_LOCKED)) { q_vector->state |= IXGBE_QV_STATE_POLL_YIELD; rc = false; -#ifdef LL_EXTENDED_STATS +#ifdef BP_EXTENDED_STATS q_vector->rx.ring->stats.yields++; #endif } else @@ -451,17 +474,32 @@ static inline bool ixgbe_qv_unlock_poll(struct ixgbe_q_vector *q_vector) if (q_vector->state & IXGBE_QV_STATE_POLL_YIELD) rc = true; - q_vector->state = IXGBE_QV_STATE_IDLE; + /* will reset state to idle, unless QV is disabled */ + q_vector->state &= IXGBE_QV_STATE_DISABLED; spin_unlock_bh(&q_vector->lock); return rc; } /* true if a socket is polling, even if it did not get the lock */ -static inline bool ixgbe_qv_ll_polling(struct ixgbe_q_vector *q_vector) +static inline bool ixgbe_qv_busy_polling(struct ixgbe_q_vector *q_vector) { - WARN_ON(!(q_vector->state & IXGBE_QV_LOCKED)); + WARN_ON(!(q_vector->state & IXGBE_QV_OWNED)); return q_vector->state & IXGBE_QV_USER_PEND; } + +/* false if QV is currently owned */ +static inline bool ixgbe_qv_disable(struct ixgbe_q_vector *q_vector) +{ + int rc = true; + spin_lock_bh(&q_vector->lock); + if (q_vector->state & IXGBE_QV_OWNED) + rc = false; + q_vector->state |= IXGBE_QV_STATE_DISABLED; + spin_unlock_bh(&q_vector->lock); + + return rc; +} + #else /* CONFIG_NET_RX_BUSY_POLL */ static inline void ixgbe_qv_init_lock(struct ixgbe_q_vector *q_vector) { @@ -487,10 +525,16 @@ static inline bool ixgbe_qv_unlock_poll(struct ixgbe_q_vector *q_vector) return false; } -static inline bool ixgbe_qv_ll_polling(struct ixgbe_q_vector *q_vector) +static inline bool ixgbe_qv_busy_polling(struct ixgbe_q_vector *q_vector) { return false; } + +static inline bool ixgbe_qv_disable(struct ixgbe_q_vector *q_vector) +{ + return true; +} + #endif /* CONFIG_NET_RX_BUSY_POLL */ #ifdef CONFIG_IXGBE_HWMON @@ -738,6 +782,7 @@ struct ixgbe_adapter { #endif /*CONFIG_DEBUG_FS*/ u8 default_up; + unsigned long fwd_bitmask; /* Bitmask indicating in use pools */ }; struct ixgbe_fdir_filter { @@ -786,93 +831,89 @@ extern const char ixgbe_driver_version[]; extern char ixgbe_default_device_descr[]; #endif /* IXGBE_FCOE */ -extern void ixgbe_up(struct ixgbe_adapter *adapter); -extern void ixgbe_down(struct ixgbe_adapter *adapter); -extern void ixgbe_reinit_locked(struct ixgbe_adapter *adapter); -extern void ixgbe_reset(struct ixgbe_adapter *adapter); -extern void ixgbe_set_ethtool_ops(struct net_device *netdev); -extern int ixgbe_setup_rx_resources(struct ixgbe_ring *); -extern int ixgbe_setup_tx_resources(struct ixgbe_ring *); -extern void ixgbe_free_rx_resources(struct ixgbe_ring *); -extern void ixgbe_free_tx_resources(struct ixgbe_ring *); -extern void ixgbe_configure_rx_ring(struct ixgbe_adapter *,struct ixgbe_ring *); -extern void ixgbe_configure_tx_ring(struct ixgbe_adapter *,struct ixgbe_ring *); -extern void ixgbe_disable_rx_queue(struct ixgbe_adapter *adapter, - struct ixgbe_ring *); -extern void ixgbe_update_stats(struct ixgbe_adapter *adapter); -extern int ixgbe_init_interrupt_scheme(struct ixgbe_adapter *adapter); -extern int ixgbe_wol_supported(struct ixgbe_adapter *adapter, u16 device_id, +void ixgbe_up(struct ixgbe_adapter *adapter); +void ixgbe_down(struct ixgbe_adapter *adapter); +void ixgbe_reinit_locked(struct ixgbe_adapter *adapter); +void ixgbe_reset(struct ixgbe_adapter *adapter); +void ixgbe_set_ethtool_ops(struct net_device *netdev); +int ixgbe_setup_rx_resources(struct ixgbe_ring *); +int ixgbe_setup_tx_resources(struct ixgbe_ring *); +void ixgbe_free_rx_resources(struct ixgbe_ring *); +void ixgbe_free_tx_resources(struct ixgbe_ring *); +void ixgbe_configure_rx_ring(struct ixgbe_adapter *, struct ixgbe_ring *); +void ixgbe_configure_tx_ring(struct ixgbe_adapter *, struct ixgbe_ring *); +void ixgbe_disable_rx_queue(struct ixgbe_adapter *adapter, struct ixgbe_ring *); +void ixgbe_update_stats(struct ixgbe_adapter *adapter); +int ixgbe_init_interrupt_scheme(struct ixgbe_adapter *adapter); +int ixgbe_wol_supported(struct ixgbe_adapter *adapter, u16 device_id, u16 subdevice_id); -extern void ixgbe_clear_interrupt_scheme(struct ixgbe_adapter *adapter); -extern netdev_tx_t ixgbe_xmit_frame_ring(struct sk_buff *, - struct ixgbe_adapter *, - struct ixgbe_ring *); -extern void ixgbe_unmap_and_free_tx_resource(struct ixgbe_ring *, - struct ixgbe_tx_buffer *); -extern void ixgbe_alloc_rx_buffers(struct ixgbe_ring *, u16); -extern void ixgbe_write_eitr(struct ixgbe_q_vector *); -extern int ixgbe_poll(struct napi_struct *napi, int budget); -extern int ethtool_ioctl(struct ifreq *ifr); -extern s32 ixgbe_reinit_fdir_tables_82599(struct ixgbe_hw *hw); -extern s32 ixgbe_init_fdir_signature_82599(struct ixgbe_hw *hw, u32 fdirctrl); -extern s32 ixgbe_init_fdir_perfect_82599(struct ixgbe_hw *hw, u32 fdirctrl); -extern s32 ixgbe_fdir_add_signature_filter_82599(struct ixgbe_hw *hw, - union ixgbe_atr_hash_dword input, - union ixgbe_atr_hash_dword common, - u8 queue); -extern s32 ixgbe_fdir_set_input_mask_82599(struct ixgbe_hw *hw, - union ixgbe_atr_input *input_mask); -extern s32 ixgbe_fdir_write_perfect_filter_82599(struct ixgbe_hw *hw, - union ixgbe_atr_input *input, - u16 soft_id, u8 queue); -extern s32 ixgbe_fdir_erase_perfect_filter_82599(struct ixgbe_hw *hw, - union ixgbe_atr_input *input, - u16 soft_id); -extern void ixgbe_atr_compute_perfect_hash_82599(union ixgbe_atr_input *input, - union ixgbe_atr_input *mask); -extern bool ixgbe_verify_lesm_fw_enabled_82599(struct ixgbe_hw *hw); -extern void ixgbe_set_rx_mode(struct net_device *netdev); +void ixgbe_clear_interrupt_scheme(struct ixgbe_adapter *adapter); +netdev_tx_t ixgbe_xmit_frame_ring(struct sk_buff *, struct ixgbe_adapter *, + struct ixgbe_ring *); +void ixgbe_unmap_and_free_tx_resource(struct ixgbe_ring *, + struct ixgbe_tx_buffer *); +void ixgbe_alloc_rx_buffers(struct ixgbe_ring *, u16); +void ixgbe_write_eitr(struct ixgbe_q_vector *); +int ixgbe_poll(struct napi_struct *napi, int budget); +int ethtool_ioctl(struct ifreq *ifr); +s32 ixgbe_reinit_fdir_tables_82599(struct ixgbe_hw *hw); +s32 ixgbe_init_fdir_signature_82599(struct ixgbe_hw *hw, u32 fdirctrl); +s32 ixgbe_init_fdir_perfect_82599(struct ixgbe_hw *hw, u32 fdirctrl); +s32 ixgbe_fdir_add_signature_filter_82599(struct ixgbe_hw *hw, + union ixgbe_atr_hash_dword input, + union ixgbe_atr_hash_dword common, + u8 queue); +s32 ixgbe_fdir_set_input_mask_82599(struct ixgbe_hw *hw, + union ixgbe_atr_input *input_mask); +s32 ixgbe_fdir_write_perfect_filter_82599(struct ixgbe_hw *hw, + union ixgbe_atr_input *input, + u16 soft_id, u8 queue); +s32 ixgbe_fdir_erase_perfect_filter_82599(struct ixgbe_hw *hw, + union ixgbe_atr_input *input, + u16 soft_id); +void ixgbe_atr_compute_perfect_hash_82599(union ixgbe_atr_input *input, + union ixgbe_atr_input *mask); +bool ixgbe_verify_lesm_fw_enabled_82599(struct ixgbe_hw *hw); +void ixgbe_set_rx_mode(struct net_device *netdev); #ifdef CONFIG_IXGBE_DCB -extern void ixgbe_set_rx_drop_en(struct ixgbe_adapter *adapter); +void ixgbe_set_rx_drop_en(struct ixgbe_adapter *adapter); #endif -extern int ixgbe_setup_tc(struct net_device *dev, u8 tc); -extern void ixgbe_tx_ctxtdesc(struct ixgbe_ring *, u32, u32, u32, u32); -extern void ixgbe_do_reset(struct net_device *netdev); +int ixgbe_setup_tc(struct net_device *dev, u8 tc); +void ixgbe_tx_ctxtdesc(struct ixgbe_ring *, u32, u32, u32, u32); +void ixgbe_do_reset(struct net_device *netdev); #ifdef CONFIG_IXGBE_HWMON -extern void ixgbe_sysfs_exit(struct ixgbe_adapter *adapter); -extern int ixgbe_sysfs_init(struct ixgbe_adapter *adapter); +void ixgbe_sysfs_exit(struct ixgbe_adapter *adapter); +int ixgbe_sysfs_init(struct ixgbe_adapter *adapter); #endif /* CONFIG_IXGBE_HWMON */ #ifdef IXGBE_FCOE -extern void ixgbe_configure_fcoe(struct ixgbe_adapter *adapter); -extern int ixgbe_fso(struct ixgbe_ring *tx_ring, - struct ixgbe_tx_buffer *first, - u8 *hdr_len); -extern int ixgbe_fcoe_ddp(struct ixgbe_adapter *adapter, - union ixgbe_adv_rx_desc *rx_desc, - struct sk_buff *skb); -extern int ixgbe_fcoe_ddp_get(struct net_device *netdev, u16 xid, - struct scatterlist *sgl, unsigned int sgc); -extern int ixgbe_fcoe_ddp_target(struct net_device *netdev, u16 xid, - struct scatterlist *sgl, unsigned int sgc); -extern int ixgbe_fcoe_ddp_put(struct net_device *netdev, u16 xid); -extern int ixgbe_setup_fcoe_ddp_resources(struct ixgbe_adapter *adapter); -extern void ixgbe_free_fcoe_ddp_resources(struct ixgbe_adapter *adapter); -extern int ixgbe_fcoe_enable(struct net_device *netdev); -extern int ixgbe_fcoe_disable(struct net_device *netdev); +void ixgbe_configure_fcoe(struct ixgbe_adapter *adapter); +int ixgbe_fso(struct ixgbe_ring *tx_ring, struct ixgbe_tx_buffer *first, + u8 *hdr_len); +int ixgbe_fcoe_ddp(struct ixgbe_adapter *adapter, + union ixgbe_adv_rx_desc *rx_desc, struct sk_buff *skb); +int ixgbe_fcoe_ddp_get(struct net_device *netdev, u16 xid, + struct scatterlist *sgl, unsigned int sgc); +int ixgbe_fcoe_ddp_target(struct net_device *netdev, u16 xid, + struct scatterlist *sgl, unsigned int sgc); +int ixgbe_fcoe_ddp_put(struct net_device *netdev, u16 xid); +int ixgbe_setup_fcoe_ddp_resources(struct ixgbe_adapter *adapter); +void ixgbe_free_fcoe_ddp_resources(struct ixgbe_adapter *adapter); +int ixgbe_fcoe_enable(struct net_device *netdev); +int ixgbe_fcoe_disable(struct net_device *netdev); #ifdef CONFIG_IXGBE_DCB -extern u8 ixgbe_fcoe_getapp(struct ixgbe_adapter *adapter); -extern u8 ixgbe_fcoe_setapp(struct ixgbe_adapter *adapter, u8 up); +u8 ixgbe_fcoe_getapp(struct ixgbe_adapter *adapter); +u8 ixgbe_fcoe_setapp(struct ixgbe_adapter *adapter, u8 up); #endif /* CONFIG_IXGBE_DCB */ -extern int ixgbe_fcoe_get_wwn(struct net_device *netdev, u64 *wwn, int type); -extern int ixgbe_fcoe_get_hbainfo(struct net_device *netdev, - struct netdev_fcoe_hbainfo *info); -extern u8 ixgbe_fcoe_get_tc(struct ixgbe_adapter *adapter); +int ixgbe_fcoe_get_wwn(struct net_device *netdev, u64 *wwn, int type); +int ixgbe_fcoe_get_hbainfo(struct net_device *netdev, + struct netdev_fcoe_hbainfo *info); +u8 ixgbe_fcoe_get_tc(struct ixgbe_adapter *adapter); #endif /* IXGBE_FCOE */ #ifdef CONFIG_DEBUG_FS -extern void ixgbe_dbg_adapter_init(struct ixgbe_adapter *adapter); -extern void ixgbe_dbg_adapter_exit(struct ixgbe_adapter *adapter); -extern void ixgbe_dbg_init(void); -extern void ixgbe_dbg_exit(void); +void ixgbe_dbg_adapter_init(struct ixgbe_adapter *adapter); +void ixgbe_dbg_adapter_exit(struct ixgbe_adapter *adapter); +void ixgbe_dbg_init(void); +void ixgbe_dbg_exit(void); #else static inline void ixgbe_dbg_adapter_init(struct ixgbe_adapter *adapter) {} static inline void ixgbe_dbg_adapter_exit(struct ixgbe_adapter *adapter) {} @@ -884,12 +925,12 @@ static inline struct netdev_queue *txring_txq(const struct ixgbe_ring *ring) return netdev_get_tx_queue(ring->netdev, ring->queue_index); } -extern void ixgbe_ptp_init(struct ixgbe_adapter *adapter); -extern void ixgbe_ptp_stop(struct ixgbe_adapter *adapter); -extern void ixgbe_ptp_overflow_check(struct ixgbe_adapter *adapter); -extern void ixgbe_ptp_rx_hang(struct ixgbe_adapter *adapter); -extern void __ixgbe_ptp_rx_hwtstamp(struct ixgbe_q_vector *q_vector, - struct sk_buff *skb); +void ixgbe_ptp_init(struct ixgbe_adapter *adapter); +void ixgbe_ptp_stop(struct ixgbe_adapter *adapter); +void ixgbe_ptp_overflow_check(struct ixgbe_adapter *adapter); +void ixgbe_ptp_rx_hang(struct ixgbe_adapter *adapter); +void __ixgbe_ptp_rx_hwtstamp(struct ixgbe_q_vector *q_vector, + struct sk_buff *skb); static inline void ixgbe_ptp_rx_hwtstamp(struct ixgbe_ring *rx_ring, union ixgbe_adv_rx_desc *rx_desc, struct sk_buff *skb) @@ -906,13 +947,16 @@ static inline void ixgbe_ptp_rx_hwtstamp(struct ixgbe_ring *rx_ring, rx_ring->last_rx_timestamp = jiffies; } -extern int ixgbe_ptp_hwtstamp_ioctl(struct ixgbe_adapter *adapter, - struct ifreq *ifr, int cmd); -extern void ixgbe_ptp_start_cyclecounter(struct ixgbe_adapter *adapter); -extern void ixgbe_ptp_reset(struct ixgbe_adapter *adapter); -extern void ixgbe_ptp_check_pps_event(struct ixgbe_adapter *adapter, u32 eicr); +int ixgbe_ptp_hwtstamp_ioctl(struct ixgbe_adapter *adapter, struct ifreq *ifr, + int cmd); +void ixgbe_ptp_start_cyclecounter(struct ixgbe_adapter *adapter); +void ixgbe_ptp_reset(struct ixgbe_adapter *adapter); +void ixgbe_ptp_check_pps_event(struct ixgbe_adapter *adapter, u32 eicr); #ifdef CONFIG_PCI_IOV void ixgbe_sriov_reinit(struct ixgbe_adapter *adapter); #endif +netdev_tx_t ixgbe_xmit_frame_ring(struct sk_buff *skb, + struct ixgbe_adapter *adapter, + struct ixgbe_ring *tx_ring); #endif /* _IXGBE_H_ */ diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c index e8649ab..4e7c9b0 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c @@ -442,7 +442,7 @@ static void ixgbe_set_msglevel(struct net_device *netdev, u32 data) static int ixgbe_get_regs_len(struct net_device *netdev) { -#define IXGBE_REGS_LEN 1129 +#define IXGBE_REGS_LEN 1139 return IXGBE_REGS_LEN * sizeof(u32); } @@ -602,22 +602,53 @@ static void ixgbe_get_regs(struct net_device *netdev, regs_buff[828] = IXGBE_READ_REG(hw, IXGBE_FHFT(0)); /* DCB */ - regs_buff[829] = IXGBE_READ_REG(hw, IXGBE_RMCS); - regs_buff[830] = IXGBE_READ_REG(hw, IXGBE_DPMCS); - regs_buff[831] = IXGBE_READ_REG(hw, IXGBE_PDPMCS); - regs_buff[832] = IXGBE_READ_REG(hw, IXGBE_RUPPBMR); - for (i = 0; i < 8; i++) - regs_buff[833 + i] = IXGBE_READ_REG(hw, IXGBE_RT2CR(i)); - for (i = 0; i < 8; i++) - regs_buff[841 + i] = IXGBE_READ_REG(hw, IXGBE_RT2SR(i)); - for (i = 0; i < 8; i++) - regs_buff[849 + i] = IXGBE_READ_REG(hw, IXGBE_TDTQ2TCCR(i)); - for (i = 0; i < 8; i++) - regs_buff[857 + i] = IXGBE_READ_REG(hw, IXGBE_TDTQ2TCSR(i)); + regs_buff[829] = IXGBE_READ_REG(hw, IXGBE_RMCS); /* same as FCCFG */ + regs_buff[831] = IXGBE_READ_REG(hw, IXGBE_PDPMCS); /* same as RTTPCS */ + + switch (hw->mac.type) { + case ixgbe_mac_82598EB: + regs_buff[830] = IXGBE_READ_REG(hw, IXGBE_DPMCS); + regs_buff[832] = IXGBE_READ_REG(hw, IXGBE_RUPPBMR); + for (i = 0; i < 8; i++) + regs_buff[833 + i] = + IXGBE_READ_REG(hw, IXGBE_RT2CR(i)); + for (i = 0; i < 8; i++) + regs_buff[841 + i] = + IXGBE_READ_REG(hw, IXGBE_RT2SR(i)); + for (i = 0; i < 8; i++) + regs_buff[849 + i] = + IXGBE_READ_REG(hw, IXGBE_TDTQ2TCCR(i)); + for (i = 0; i < 8; i++) + regs_buff[857 + i] = + IXGBE_READ_REG(hw, IXGBE_TDTQ2TCSR(i)); + break; + case ixgbe_mac_82599EB: + case ixgbe_mac_X540: + regs_buff[830] = IXGBE_READ_REG(hw, IXGBE_RTTDCS); + regs_buff[832] = IXGBE_READ_REG(hw, IXGBE_RTRPCS); + for (i = 0; i < 8; i++) + regs_buff[833 + i] = + IXGBE_READ_REG(hw, IXGBE_RTRPT4C(i)); + for (i = 0; i < 8; i++) + regs_buff[841 + i] = + IXGBE_READ_REG(hw, IXGBE_RTRPT4S(i)); + for (i = 0; i < 8; i++) + regs_buff[849 + i] = + IXGBE_READ_REG(hw, IXGBE_RTTDT2C(i)); + for (i = 0; i < 8; i++) + regs_buff[857 + i] = + IXGBE_READ_REG(hw, IXGBE_RTTDT2S(i)); + break; + default: + break; + } + for (i = 0; i < 8; i++) - regs_buff[865 + i] = IXGBE_READ_REG(hw, IXGBE_TDPT2TCCR(i)); + regs_buff[865 + i] = + IXGBE_READ_REG(hw, IXGBE_TDPT2TCCR(i)); /* same as RTTPT2C */ for (i = 0; i < 8; i++) - regs_buff[873 + i] = IXGBE_READ_REG(hw, IXGBE_TDPT2TCSR(i)); + regs_buff[873 + i] = + IXGBE_READ_REG(hw, IXGBE_TDPT2TCSR(i)); /* same as RTTPT2S */ /* Statistics */ regs_buff[881] = IXGBE_GET_STAT(adapter, crcerrs); @@ -757,6 +788,20 @@ static void ixgbe_get_regs(struct net_device *netdev, /* 82599 X540 specific registers */ regs_buff[1128] = IXGBE_READ_REG(hw, IXGBE_MFLCN); + + /* 82599 X540 specific DCB registers */ + regs_buff[1129] = IXGBE_READ_REG(hw, IXGBE_RTRUP2TC); + regs_buff[1130] = IXGBE_READ_REG(hw, IXGBE_RTTUP2TC); + for (i = 0; i < 4; i++) + regs_buff[1131 + i] = IXGBE_READ_REG(hw, IXGBE_TXLLQ(i)); + regs_buff[1135] = IXGBE_READ_REG(hw, IXGBE_RTTBCNRM); + /* same as RTTQCNRM */ + regs_buff[1136] = IXGBE_READ_REG(hw, IXGBE_RTTBCNRD); + /* same as RTTQCNRR */ + + /* X540 specific DCB registers */ + regs_buff[1137] = IXGBE_READ_REG(hw, IXGBE_RTTQCNCR); + regs_buff[1138] = IXGBE_READ_REG(hw, IXGBE_RTTQCNTG); } static int ixgbe_get_eeprom_len(struct net_device *netdev) @@ -1072,7 +1117,7 @@ static void ixgbe_get_ethtool_stats(struct net_device *netdev, data[i] = 0; data[i+1] = 0; i += 2; -#ifdef LL_EXTENDED_STATS +#ifdef BP_EXTENDED_STATS data[i] = 0; data[i+1] = 0; data[i+2] = 0; @@ -1087,7 +1132,7 @@ static void ixgbe_get_ethtool_stats(struct net_device *netdev, data[i+1] = ring->stats.bytes; } while (u64_stats_fetch_retry_bh(&ring->syncp, start)); i += 2; -#ifdef LL_EXTENDED_STATS +#ifdef BP_EXTENDED_STATS data[i] = ring->stats.yields; data[i+1] = ring->stats.misses; data[i+2] = ring->stats.cleaned; @@ -1100,7 +1145,7 @@ static void ixgbe_get_ethtool_stats(struct net_device *netdev, data[i] = 0; data[i+1] = 0; i += 2; -#ifdef LL_EXTENDED_STATS +#ifdef BP_EXTENDED_STATS data[i] = 0; data[i+1] = 0; data[i+2] = 0; @@ -1115,7 +1160,7 @@ static void ixgbe_get_ethtool_stats(struct net_device *netdev, data[i+1] = ring->stats.bytes; } while (u64_stats_fetch_retry_bh(&ring->syncp, start)); i += 2; -#ifdef LL_EXTENDED_STATS +#ifdef BP_EXTENDED_STATS data[i] = ring->stats.yields; data[i+1] = ring->stats.misses; data[i+2] = ring->stats.cleaned; @@ -1157,28 +1202,28 @@ static void ixgbe_get_strings(struct net_device *netdev, u32 stringset, p += ETH_GSTRING_LEN; sprintf(p, "tx_queue_%u_bytes", i); p += ETH_GSTRING_LEN; -#ifdef LL_EXTENDED_STATS - sprintf(p, "tx_queue_%u_ll_napi_yield", i); +#ifdef BP_EXTENDED_STATS + sprintf(p, "tx_queue_%u_bp_napi_yield", i); p += ETH_GSTRING_LEN; - sprintf(p, "tx_queue_%u_ll_misses", i); + sprintf(p, "tx_queue_%u_bp_misses", i); p += ETH_GSTRING_LEN; - sprintf(p, "tx_queue_%u_ll_cleaned", i); + sprintf(p, "tx_queue_%u_bp_cleaned", i); p += ETH_GSTRING_LEN; -#endif /* LL_EXTENDED_STATS */ +#endif /* BP_EXTENDED_STATS */ } for (i = 0; i < IXGBE_NUM_RX_QUEUES; i++) { sprintf(p, "rx_queue_%u_packets", i); p += ETH_GSTRING_LEN; sprintf(p, "rx_queue_%u_bytes", i); p += ETH_GSTRING_LEN; -#ifdef LL_EXTENDED_STATS - sprintf(p, "rx_queue_%u_ll_poll_yield", i); +#ifdef BP_EXTENDED_STATS + sprintf(p, "rx_queue_%u_bp_poll_yield", i); p += ETH_GSTRING_LEN; - sprintf(p, "rx_queue_%u_ll_misses", i); + sprintf(p, "rx_queue_%u_bp_misses", i); p += ETH_GSTRING_LEN; - sprintf(p, "rx_queue_%u_ll_cleaned", i); + sprintf(p, "rx_queue_%u_bp_cleaned", i); p += ETH_GSTRING_LEN; -#endif /* LL_EXTENDED_STATS */ +#endif /* BP_EXTENDED_STATS */ } for (i = 0; i < IXGBE_MAX_PACKET_BUFFERS; i++) { sprintf(p, "tx_pb_%u_pxon", i); @@ -2212,13 +2257,13 @@ static int ixgbe_set_coalesce(struct net_device *netdev, #if IS_ENABLED(CONFIG_BQL) /* detect ITR changes that require update of TXDCTL.WTHRESH */ - if ((adapter->tx_itr_setting > 1) && + if ((adapter->tx_itr_setting != 1) && (adapter->tx_itr_setting < IXGBE_100K_ITR)) { if ((tx_itr_prev == 1) || - (tx_itr_prev > IXGBE_100K_ITR)) + (tx_itr_prev >= IXGBE_100K_ITR)) need_reset = true; } else { - if ((tx_itr_prev > 1) && + if ((tx_itr_prev != 1) && (tx_itr_prev < IXGBE_100K_ITR)) need_reset = true; } diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c index 90b4e10..32e3eaa 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c @@ -498,6 +498,7 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter) #ifdef IXGBE_FCOE u16 fcoe_i = 0; #endif + bool pools = (find_first_zero_bit(&adapter->fwd_bitmask, 32) > 1); /* only proceed if SR-IOV is enabled */ if (!(adapter->flags & IXGBE_FLAG_SRIOV_ENABLED)) @@ -510,7 +511,7 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter) vmdq_i = min_t(u16, IXGBE_MAX_VMDQ_INDICES, vmdq_i); /* 64 pool mode with 2 queues per pool */ - if ((vmdq_i > 32) || (rss_i < 4)) { + if ((vmdq_i > 32) || (rss_i < 4) || (vmdq_i > 16 && pools)) { vmdq_m = IXGBE_82599_VMDQ_2Q_MASK; rss_m = IXGBE_RSS_2Q_MASK; rss_i = min_t(u16, rss_i, 2); @@ -852,7 +853,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter, /* apply Tx specific ring traits */ ring->count = adapter->tx_ring_count; - ring->queue_index = txr_idx; + if (adapter->num_rx_pools > 1) + ring->queue_index = + txr_idx % adapter->num_rx_queues_per_pool; + else + ring->queue_index = txr_idx; /* assign ring to adapter */ adapter->tx_ring[txr_idx] = ring; @@ -895,7 +900,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter, #endif /* IXGBE_FCOE */ /* apply Rx specific ring traits */ ring->count = adapter->rx_ring_count; - ring->queue_index = rxr_idx; + if (adapter->num_rx_pools > 1) + ring->queue_index = + rxr_idx % adapter->num_rx_queues_per_pool; + else + ring->queue_index = rxr_idx; /* assign ring to adapter */ adapter->rx_ring[rxr_idx] = ring; diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c index 0ade0cd..bd8f523 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c @@ -44,6 +44,7 @@ #include <linux/ethtool.h> #include <linux/if.h> #include <linux/if_vlan.h> +#include <linux/if_macvlan.h> #include <linux/if_bridge.h> #include <linux/prefetch.h> #include <scsi/fc/fc_fcoe.h> @@ -132,7 +133,7 @@ static struct notifier_block dca_notifier = { static unsigned int max_vfs; module_param(max_vfs, uint, 0); MODULE_PARM_DESC(max_vfs, - "Maximum number of virtual functions to allocate per physical function - default is zero and maximum value is 63"); + "Maximum number of virtual functions to allocate per physical function - default is zero and maximum value is 63. (Deprecated)"); #endif /* CONFIG_PCI_IOV */ static unsigned int allow_unsupported_sfp; @@ -153,7 +154,6 @@ MODULE_VERSION(DRV_VERSION); static int ixgbe_read_pci_cfg_word_parent(struct ixgbe_adapter *adapter, u32 reg, u16 *value) { - int pos = 0; struct pci_dev *parent_dev; struct pci_bus *parent_bus; @@ -165,11 +165,10 @@ static int ixgbe_read_pci_cfg_word_parent(struct ixgbe_adapter *adapter, if (!parent_dev) return -1; - pos = pci_find_capability(parent_dev, PCI_CAP_ID_EXP); - if (!pos) + if (!pci_is_pcie(parent_dev)) return -1; - pci_read_config_word(parent_dev, pos + reg, value); + pcie_capability_read_word(parent_dev, reg, value); return 0; } @@ -247,7 +246,7 @@ static void ixgbe_check_minimum_link(struct ixgbe_adapter *adapter, max_gts = 4 * width; break; case PCIE_SPEED_8_0GT: - /* 128b/130b encoding only reduces throughput by 1% */ + /* 128b/130b encoding reduces throughput by less than 2% */ max_gts = 8 * width; break; default: @@ -265,7 +264,7 @@ static void ixgbe_check_minimum_link(struct ixgbe_adapter *adapter, width, (speed == PCIE_SPEED_2_5GT ? "20%" : speed == PCIE_SPEED_5_0GT ? "20%" : - speed == PCIE_SPEED_8_0GT ? "N/a" : + speed == PCIE_SPEED_8_0GT ? "<2%" : "Unknown")); if (max_gts < expected_gts) { @@ -872,11 +871,18 @@ static u64 ixgbe_get_tx_completed(struct ixgbe_ring *ring) static u64 ixgbe_get_tx_pending(struct ixgbe_ring *ring) { - struct ixgbe_adapter *adapter = netdev_priv(ring->netdev); - struct ixgbe_hw *hw = &adapter->hw; + struct ixgbe_adapter *adapter; + struct ixgbe_hw *hw; + u32 head, tail; - u32 head = IXGBE_READ_REG(hw, IXGBE_TDH(ring->reg_idx)); - u32 tail = IXGBE_READ_REG(hw, IXGBE_TDT(ring->reg_idx)); + if (ring->l2_accel_priv) + adapter = ring->l2_accel_priv->real_adapter; + else + adapter = netdev_priv(ring->netdev); + + hw = &adapter->hw; + head = IXGBE_READ_REG(hw, IXGBE_TDH(ring->reg_idx)); + tail = IXGBE_READ_REG(hw, IXGBE_TDT(ring->reg_idx)); if (head != tail) return (head < tail) ? @@ -1585,7 +1591,7 @@ static void ixgbe_rx_skb(struct ixgbe_q_vector *q_vector, { struct ixgbe_adapter *adapter = q_vector->adapter; - if (ixgbe_qv_ll_polling(q_vector)) + if (ixgbe_qv_busy_polling(q_vector)) netif_receive_skb(skb); else if (!(adapter->flags & IXGBE_FLAG_IN_NETPOLL)) napi_gro_receive(&q_vector->napi, skb); @@ -2097,7 +2103,7 @@ static int ixgbe_low_latency_recv(struct napi_struct *napi) ixgbe_for_each_ring(ring, q_vector->rx) { found = ixgbe_clean_rx_irq(q_vector, ring, 4); -#ifdef LL_EXTENDED_STATS +#ifdef BP_EXTENDED_STATS if (found) ring->stats.cleaned += found; else @@ -3005,7 +3011,7 @@ void ixgbe_configure_tx_ring(struct ixgbe_adapter *adapter, struct ixgbe_q_vector *q_vector = ring->q_vector; if (q_vector) - netif_set_xps_queue(adapter->netdev, + netif_set_xps_queue(ring->netdev, &q_vector->affinity_mask, ring->queue_index); } @@ -3395,7 +3401,7 @@ static void ixgbe_setup_psrtype(struct ixgbe_adapter *adapter) { struct ixgbe_hw *hw = &adapter->hw; int rss_i = adapter->ring_feature[RING_F_RSS].indices; - int p; + u16 pool; /* PSRTYPE must be initialized in non 82598 adapters */ u32 psrtype = IXGBE_PSRTYPE_TCPHDR | @@ -3412,9 +3418,8 @@ static void ixgbe_setup_psrtype(struct ixgbe_adapter *adapter) else if (rss_i > 1) psrtype |= 1 << 29; - for (p = 0; p < adapter->num_rx_pools; p++) - IXGBE_WRITE_REG(hw, IXGBE_PSRTYPE(VMDQ_P(p)), - psrtype); + for_each_set_bit(pool, &adapter->fwd_bitmask, 32) + IXGBE_WRITE_REG(hw, IXGBE_PSRTYPE(VMDQ_P(pool)), psrtype); } static void ixgbe_configure_virtualization(struct ixgbe_adapter *adapter) @@ -3683,7 +3688,11 @@ static void ixgbe_vlan_strip_disable(struct ixgbe_adapter *adapter) case ixgbe_mac_82599EB: case ixgbe_mac_X540: for (i = 0; i < adapter->num_rx_queues; i++) { - j = adapter->rx_ring[i]->reg_idx; + struct ixgbe_ring *ring = adapter->rx_ring[i]; + + if (ring->l2_accel_priv) + continue; + j = ring->reg_idx; vlnctrl = IXGBE_READ_REG(hw, IXGBE_RXDCTL(j)); vlnctrl &= ~IXGBE_RXDCTL_VME; IXGBE_WRITE_REG(hw, IXGBE_RXDCTL(j), vlnctrl); @@ -3713,7 +3722,11 @@ static void ixgbe_vlan_strip_enable(struct ixgbe_adapter *adapter) case ixgbe_mac_82599EB: case ixgbe_mac_X540: for (i = 0; i < adapter->num_rx_queues; i++) { - j = adapter->rx_ring[i]->reg_idx; + struct ixgbe_ring *ring = adapter->rx_ring[i]; + + if (ring->l2_accel_priv) + continue; + j = ring->reg_idx; vlnctrl = IXGBE_READ_REG(hw, IXGBE_RXDCTL(j)); vlnctrl |= IXGBE_RXDCTL_VME; IXGBE_WRITE_REG(hw, IXGBE_RXDCTL(j), vlnctrl); @@ -3750,7 +3763,7 @@ static int ixgbe_write_uc_addr_list(struct net_device *netdev) unsigned int rar_entries = hw->mac.num_rar_entries - 1; int count = 0; - /* In SR-IOV mode significantly less RAR entries are available */ + /* In SR-IOV/VMDQ modes significantly less RAR entries are available */ if (adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) rar_entries = IXGBE_MAX_PF_MACVLANS - 1; @@ -3825,14 +3838,6 @@ void ixgbe_set_rx_mode(struct net_device *netdev) if (netdev->flags & IFF_ALLMULTI) { fctrl |= IXGBE_FCTRL_MPE; vmolr |= IXGBE_VMOLR_MPE; - } else { - /* - * Write addresses to the MTA, if the attempt fails - * then we should just turn on promiscuous mode so - * that we can at least receive multicast traffic - */ - hw->mac.ops.update_mc_addr_list(hw, netdev); - vmolr |= IXGBE_VMOLR_ROMPE; } ixgbe_vlan_filter_enable(adapter); hw->addr_ctrl.user_set_promisc = false; @@ -3849,6 +3854,13 @@ void ixgbe_set_rx_mode(struct net_device *netdev) vmolr |= IXGBE_VMOLR_ROPE; } + /* Write addresses to the MTA, if the attempt fails + * then we should just turn on promiscuous mode so + * that we can at least receive multicast traffic + */ + hw->mac.ops.update_mc_addr_list(hw, netdev); + vmolr |= IXGBE_VMOLR_ROMPE; + if (adapter->num_vfs) ixgbe_restore_vf_multicasts(adapter); @@ -3893,15 +3905,13 @@ static void ixgbe_napi_disable_all(struct ixgbe_adapter *adapter) { int q_idx; - local_bh_disable(); /* for ixgbe_qv_lock_napi() */ for (q_idx = 0; q_idx < adapter->num_q_vectors; q_idx++) { napi_disable(&adapter->q_vector[q_idx]->napi); - while (!ixgbe_qv_lock_napi(adapter->q_vector[q_idx])) { + while (!ixgbe_qv_disable(adapter->q_vector[q_idx])) { pr_info("QV %d locked\n", q_idx); - mdelay(1); + usleep_range(1000, 20000); } } - local_bh_enable(); } #ifdef CONFIG_IXGBE_DCB @@ -4118,6 +4128,228 @@ static void ixgbe_fdir_filter_restore(struct ixgbe_adapter *adapter) spin_unlock(&adapter->fdir_perfect_lock); } +static void ixgbe_macvlan_set_rx_mode(struct net_device *dev, unsigned int pool, + struct ixgbe_adapter *adapter) +{ + struct ixgbe_hw *hw = &adapter->hw; + u32 vmolr; + + /* No unicast promiscuous support for VMDQ devices. */ + vmolr = IXGBE_READ_REG(hw, IXGBE_VMOLR(pool)); + vmolr |= (IXGBE_VMOLR_ROMPE | IXGBE_VMOLR_BAM | IXGBE_VMOLR_AUPE); + + /* clear the affected bit */ + vmolr &= ~IXGBE_VMOLR_MPE; + + if (dev->flags & IFF_ALLMULTI) { + vmolr |= IXGBE_VMOLR_MPE; + } else { + vmolr |= IXGBE_VMOLR_ROMPE; + hw->mac.ops.update_mc_addr_list(hw, dev); + } + ixgbe_write_uc_addr_list(adapter->netdev); + IXGBE_WRITE_REG(hw, IXGBE_VMOLR(pool), vmolr); +} + +static void ixgbe_add_mac_filter(struct ixgbe_adapter *adapter, + u8 *addr, u16 pool) +{ + struct ixgbe_hw *hw = &adapter->hw; + unsigned int entry; + + entry = hw->mac.num_rar_entries - pool; + hw->mac.ops.set_rar(hw, entry, addr, VMDQ_P(pool), IXGBE_RAH_AV); +} + +static void ixgbe_fwd_psrtype(struct ixgbe_fwd_adapter *vadapter) +{ + struct ixgbe_adapter *adapter = vadapter->real_adapter; + int rss_i = adapter->num_rx_queues_per_pool; + struct ixgbe_hw *hw = &adapter->hw; + u16 pool = vadapter->pool; + u32 psrtype = IXGBE_PSRTYPE_TCPHDR | + IXGBE_PSRTYPE_UDPHDR | + IXGBE_PSRTYPE_IPV4HDR | + IXGBE_PSRTYPE_L2HDR | + IXGBE_PSRTYPE_IPV6HDR; + + if (hw->mac.type == ixgbe_mac_82598EB) + return; + + if (rss_i > 3) + psrtype |= 2 << 29; + else if (rss_i > 1) + psrtype |= 1 << 29; + + IXGBE_WRITE_REG(hw, IXGBE_PSRTYPE(VMDQ_P(pool)), psrtype); +} + +/** + * ixgbe_clean_rx_ring - Free Rx Buffers per Queue + * @rx_ring: ring to free buffers from + **/ +static void ixgbe_clean_rx_ring(struct ixgbe_ring *rx_ring) +{ + struct device *dev = rx_ring->dev; + unsigned long size; + u16 i; + + /* ring already cleared, nothing to do */ + if (!rx_ring->rx_buffer_info) + return; + + /* Free all the Rx ring sk_buffs */ + for (i = 0; i < rx_ring->count; i++) { + struct ixgbe_rx_buffer *rx_buffer; + + rx_buffer = &rx_ring->rx_buffer_info[i]; + if (rx_buffer->skb) { + struct sk_buff *skb = rx_buffer->skb; + if (IXGBE_CB(skb)->page_released) { + dma_unmap_page(dev, + IXGBE_CB(skb)->dma, + ixgbe_rx_bufsz(rx_ring), + DMA_FROM_DEVICE); + IXGBE_CB(skb)->page_released = false; + } + dev_kfree_skb(skb); + } + rx_buffer->skb = NULL; + if (rx_buffer->dma) + dma_unmap_page(dev, rx_buffer->dma, + ixgbe_rx_pg_size(rx_ring), + DMA_FROM_DEVICE); + rx_buffer->dma = 0; + if (rx_buffer->page) + __free_pages(rx_buffer->page, + ixgbe_rx_pg_order(rx_ring)); + rx_buffer->page = NULL; + } + + size = sizeof(struct ixgbe_rx_buffer) * rx_ring->count; + memset(rx_ring->rx_buffer_info, 0, size); + + /* Zero out the descriptor ring */ + memset(rx_ring->desc, 0, rx_ring->size); + + rx_ring->next_to_alloc = 0; + rx_ring->next_to_clean = 0; + rx_ring->next_to_use = 0; +} + +static void ixgbe_disable_fwd_ring(struct ixgbe_fwd_adapter *vadapter, + struct ixgbe_ring *rx_ring) +{ + struct ixgbe_adapter *adapter = vadapter->real_adapter; + int index = rx_ring->queue_index + vadapter->rx_base_queue; + + /* shutdown specific queue receive and wait for dma to settle */ + ixgbe_disable_rx_queue(adapter, rx_ring); + usleep_range(10000, 20000); + ixgbe_irq_disable_queues(adapter, ((u64)1 << index)); + ixgbe_clean_rx_ring(rx_ring); + rx_ring->l2_accel_priv = NULL; +} + +int ixgbe_fwd_ring_down(struct net_device *vdev, + struct ixgbe_fwd_adapter *accel) +{ + struct ixgbe_adapter *adapter = accel->real_adapter; + unsigned int rxbase = accel->rx_base_queue; + unsigned int txbase = accel->tx_base_queue; + int i; + + netif_tx_stop_all_queues(vdev); + + for (i = 0; i < adapter->num_rx_queues_per_pool; i++) { + ixgbe_disable_fwd_ring(accel, adapter->rx_ring[rxbase + i]); + adapter->rx_ring[rxbase + i]->netdev = adapter->netdev; + } + + for (i = 0; i < adapter->num_rx_queues_per_pool; i++) { + adapter->tx_ring[txbase + i]->l2_accel_priv = NULL; + adapter->tx_ring[txbase + i]->netdev = adapter->netdev; + } + + + return 0; +} + +static int ixgbe_fwd_ring_up(struct net_device *vdev, + struct ixgbe_fwd_adapter *accel) +{ + struct ixgbe_adapter *adapter = accel->real_adapter; + unsigned int rxbase, txbase, queues; + int i, baseq, err = 0; + + if (!test_bit(accel->pool, &adapter->fwd_bitmask)) + return 0; + + baseq = accel->pool * adapter->num_rx_queues_per_pool; + netdev_dbg(vdev, "pool %i:%i queues %i:%i VSI bitmask %lx\n", + accel->pool, adapter->num_rx_pools, + baseq, baseq + adapter->num_rx_queues_per_pool, + adapter->fwd_bitmask); + + accel->netdev = vdev; + accel->rx_base_queue = rxbase = baseq; + accel->tx_base_queue = txbase = baseq; + + for (i = 0; i < adapter->num_rx_queues_per_pool; i++) + ixgbe_disable_fwd_ring(accel, adapter->rx_ring[rxbase + i]); + + for (i = 0; i < adapter->num_rx_queues_per_pool; i++) { + adapter->rx_ring[rxbase + i]->netdev = vdev; + adapter->rx_ring[rxbase + i]->l2_accel_priv = accel; + ixgbe_configure_rx_ring(adapter, adapter->rx_ring[rxbase + i]); + } + + for (i = 0; i < adapter->num_rx_queues_per_pool; i++) { + adapter->tx_ring[txbase + i]->netdev = vdev; + adapter->tx_ring[txbase + i]->l2_accel_priv = accel; + } + + queues = min_t(unsigned int, + adapter->num_rx_queues_per_pool, vdev->num_tx_queues); + err = netif_set_real_num_tx_queues(vdev, queues); + if (err) + goto fwd_queue_err; + + err = netif_set_real_num_rx_queues(vdev, queues); + if (err) + goto fwd_queue_err; + + if (is_valid_ether_addr(vdev->dev_addr)) + ixgbe_add_mac_filter(adapter, vdev->dev_addr, accel->pool); + + ixgbe_fwd_psrtype(accel); + ixgbe_macvlan_set_rx_mode(vdev, accel->pool, adapter); + return err; +fwd_queue_err: + ixgbe_fwd_ring_down(vdev, accel); + return err; +} + +static void ixgbe_configure_dfwd(struct ixgbe_adapter *adapter) +{ + struct net_device *upper; + struct list_head *iter; + int err; + + netdev_for_each_all_upper_dev_rcu(adapter->netdev, upper, iter) { + if (netif_is_macvlan(upper)) { + struct macvlan_dev *dfwd = netdev_priv(upper); + struct ixgbe_fwd_adapter *vadapter = dfwd->fwd_priv; + + if (dfwd->fwd_priv) { + err = ixgbe_fwd_ring_up(upper, vadapter); + if (err) + continue; + } + } + } +} + static void ixgbe_configure(struct ixgbe_adapter *adapter) { struct ixgbe_hw *hw = &adapter->hw; @@ -4169,6 +4401,7 @@ static void ixgbe_configure(struct ixgbe_adapter *adapter) #endif /* IXGBE_FCOE */ ixgbe_configure_tx(adapter); ixgbe_configure_rx(adapter); + ixgbe_configure_dfwd(adapter); } static inline bool ixgbe_is_sfp(struct ixgbe_hw *hw) @@ -4322,6 +4555,8 @@ static void ixgbe_setup_gpie(struct ixgbe_adapter *adapter) static void ixgbe_up_complete(struct ixgbe_adapter *adapter) { struct ixgbe_hw *hw = &adapter->hw; + struct net_device *upper; + struct list_head *iter; int err; u32 ctrl_ext; @@ -4365,6 +4600,16 @@ static void ixgbe_up_complete(struct ixgbe_adapter *adapter) /* enable transmits */ netif_tx_start_all_queues(adapter->netdev); + /* enable any upper devices */ + netdev_for_each_all_upper_dev_rcu(adapter->netdev, upper, iter) { + if (netif_is_macvlan(upper)) { + struct macvlan_dev *vlan = netdev_priv(upper); + + if (vlan->fwd_priv) + netif_tx_start_all_queues(upper); + } + } + /* bring the link up in the watchdog, this could race with our first * link up interrupt but shouldn't be a problem */ adapter->flags |= IXGBE_FLAG_NEED_LINK_UPDATE; @@ -4456,59 +4701,6 @@ void ixgbe_reset(struct ixgbe_adapter *adapter) } /** - * ixgbe_clean_rx_ring - Free Rx Buffers per Queue - * @rx_ring: ring to free buffers from - **/ -static void ixgbe_clean_rx_ring(struct ixgbe_ring *rx_ring) -{ - struct device *dev = rx_ring->dev; - unsigned long size; - u16 i; - - /* ring already cleared, nothing to do */ - if (!rx_ring->rx_buffer_info) - return; - - /* Free all the Rx ring sk_buffs */ - for (i = 0; i < rx_ring->count; i++) { - struct ixgbe_rx_buffer *rx_buffer; - - rx_buffer = &rx_ring->rx_buffer_info[i]; - if (rx_buffer->skb) { - struct sk_buff *skb = rx_buffer->skb; - if (IXGBE_CB(skb)->page_released) { - dma_unmap_page(dev, - IXGBE_CB(skb)->dma, - ixgbe_rx_bufsz(rx_ring), - DMA_FROM_DEVICE); - IXGBE_CB(skb)->page_released = false; - } - dev_kfree_skb(skb); - } - rx_buffer->skb = NULL; - if (rx_buffer->dma) - dma_unmap_page(dev, rx_buffer->dma, - ixgbe_rx_pg_size(rx_ring), - DMA_FROM_DEVICE); - rx_buffer->dma = 0; - if (rx_buffer->page) - __free_pages(rx_buffer->page, - ixgbe_rx_pg_order(rx_ring)); - rx_buffer->page = NULL; - } - - size = sizeof(struct ixgbe_rx_buffer) * rx_ring->count; - memset(rx_ring->rx_buffer_info, 0, size); - - /* Zero out the descriptor ring */ - memset(rx_ring->desc, 0, rx_ring->size); - - rx_ring->next_to_alloc = 0; - rx_ring->next_to_clean = 0; - rx_ring->next_to_use = 0; -} - -/** * ixgbe_clean_tx_ring - Free Tx Buffers * @tx_ring: ring to be cleaned **/ @@ -4585,6 +4777,8 @@ void ixgbe_down(struct ixgbe_adapter *adapter) { struct net_device *netdev = adapter->netdev; struct ixgbe_hw *hw = &adapter->hw; + struct net_device *upper; + struct list_head *iter; u32 rxctrl; int i; @@ -4608,6 +4802,19 @@ void ixgbe_down(struct ixgbe_adapter *adapter) netif_carrier_off(netdev); netif_tx_disable(netdev); + /* disable any upper devices */ + netdev_for_each_all_upper_dev_rcu(adapter->netdev, upper, iter) { + if (netif_is_macvlan(upper)) { + struct macvlan_dev *vlan = netdev_priv(upper); + + if (vlan->fwd_priv) { + netif_tx_stop_all_queues(upper); + netif_carrier_off(upper); + netif_tx_disable(upper); + } + } + } + ixgbe_irq_disable(adapter); ixgbe_napi_disable_all(adapter); @@ -4816,11 +5023,20 @@ static int ixgbe_sw_init(struct ixgbe_adapter *adapter) hw->fc.disable_fc_autoneg = ixgbe_device_supports_autoneg_fc(hw); #ifdef CONFIG_PCI_IOV + if (max_vfs > 0) + e_dev_warn("Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.\n"); + /* assign number of SR-IOV VFs */ - if (hw->mac.type != ixgbe_mac_82598EB) - adapter->num_vfs = (max_vfs > 63) ? 0 : max_vfs; + if (hw->mac.type != ixgbe_mac_82598EB) { + if (max_vfs > 63) { + adapter->num_vfs = 0; + e_dev_warn("max_vfs parameter out of range. Not assigning any SR-IOV VFs\n"); + } else { + adapter->num_vfs = max_vfs; + } + } +#endif /* CONFIG_PCI_IOV */ -#endif /* enable itr by default in dynamic mode */ adapter->rx_itr_setting = 1; adapter->tx_itr_setting = 1; @@ -4838,6 +5054,8 @@ static int ixgbe_sw_init(struct ixgbe_adapter *adapter) return -EIO; } + /* PF holds first pool slot */ + set_bit(0, &adapter->fwd_bitmask); set_bit(__IXGBE_DOWN, &adapter->state); return 0; @@ -5143,7 +5361,7 @@ static int ixgbe_change_mtu(struct net_device *netdev, int new_mtu) static int ixgbe_open(struct net_device *netdev) { struct ixgbe_adapter *adapter = netdev_priv(netdev); - int err; + int err, queues; /* disallow open during test */ if (test_bit(__IXGBE_TESTING, &adapter->state)) @@ -5168,16 +5386,21 @@ static int ixgbe_open(struct net_device *netdev) goto err_req_irq; /* Notify the stack of the actual queue counts. */ - err = netif_set_real_num_tx_queues(netdev, - adapter->num_rx_pools > 1 ? 1 : - adapter->num_tx_queues); + if (adapter->num_rx_pools > 1) + queues = adapter->num_rx_queues_per_pool; + else + queues = adapter->num_tx_queues; + + err = netif_set_real_num_tx_queues(netdev, queues); if (err) goto err_set_queues; - - err = netif_set_real_num_rx_queues(netdev, - adapter->num_rx_pools > 1 ? 1 : - adapter->num_rx_queues); + if (adapter->num_rx_pools > 1 && + adapter->num_rx_queues > IXGBE_MAX_L2A_QUEUES) + queues = IXGBE_MAX_L2A_QUEUES; + else + queues = adapter->num_rx_queues; + err = netif_set_real_num_rx_queues(netdev, queues); if (err) goto err_set_queues; @@ -6767,8 +6990,9 @@ out_drop: return NETDEV_TX_OK; } -static netdev_tx_t ixgbe_xmit_frame(struct sk_buff *skb, - struct net_device *netdev) +static netdev_tx_t __ixgbe_xmit_frame(struct sk_buff *skb, + struct net_device *netdev, + struct ixgbe_ring *ring) { struct ixgbe_adapter *adapter = netdev_priv(netdev); struct ixgbe_ring *tx_ring; @@ -6784,10 +7008,17 @@ static netdev_tx_t ixgbe_xmit_frame(struct sk_buff *skb, skb_set_tail_pointer(skb, 17); } - tx_ring = adapter->tx_ring[skb->queue_mapping]; + tx_ring = ring ? ring : adapter->tx_ring[skb->queue_mapping]; + return ixgbe_xmit_frame_ring(skb, adapter, tx_ring); } +static netdev_tx_t ixgbe_xmit_frame(struct sk_buff *skb, + struct net_device *netdev) +{ + return __ixgbe_xmit_frame(skb, netdev, NULL); +} + /** * ixgbe_set_mac - Change the Ethernet Address of the NIC * @netdev: network interface device structure @@ -7044,6 +7275,7 @@ int ixgbe_setup_tc(struct net_device *dev, u8 tc) { struct ixgbe_adapter *adapter = netdev_priv(dev); struct ixgbe_hw *hw = &adapter->hw; + bool pools; /* Hardware supports up to 8 traffic classes */ if (tc > adapter->dcb_cfg.num_tcs.pg_tcs || @@ -7051,6 +7283,10 @@ int ixgbe_setup_tc(struct net_device *dev, u8 tc) tc < MAX_TRAFFIC_CLASS)) return -EINVAL; + pools = (find_first_zero_bit(&adapter->fwd_bitmask, 32) > 1); + if (tc && pools && adapter->num_rx_pools > IXGBE_MAX_DCBMACVLANS) + return -EBUSY; + /* Hardware has to reinitialize queues and interrupts to * match packet buffer alignment. Unfortunately, the * hardware is not flexible enough to do this dynamically. @@ -7305,6 +7541,104 @@ static int ixgbe_ndo_bridge_getlink(struct sk_buff *skb, u32 pid, u32 seq, return ndo_dflt_bridge_getlink(skb, pid, seq, dev, mode); } +static void *ixgbe_fwd_add(struct net_device *pdev, struct net_device *vdev) +{ + struct ixgbe_fwd_adapter *fwd_adapter = NULL; + struct ixgbe_adapter *adapter = netdev_priv(pdev); + unsigned int limit; + int pool, err; + +#ifdef CONFIG_RPS + if (vdev->num_rx_queues != vdev->num_tx_queues) { + netdev_info(pdev, "%s: Only supports a single queue count for TX and RX\n", + vdev->name); + return ERR_PTR(-EINVAL); + } +#endif + /* Check for hardware restriction on number of rx/tx queues */ + if (vdev->num_tx_queues > IXGBE_MAX_L2A_QUEUES || + vdev->num_tx_queues == IXGBE_BAD_L2A_QUEUE) { + netdev_info(pdev, + "%s: Supports RX/TX Queue counts 1,2, and 4\n", + pdev->name); + return ERR_PTR(-EINVAL); + } + + if (((adapter->flags & IXGBE_FLAG_DCB_ENABLED) && + adapter->num_rx_pools > IXGBE_MAX_DCBMACVLANS - 1) || + (adapter->num_rx_pools > IXGBE_MAX_MACVLANS)) + return ERR_PTR(-EBUSY); + + fwd_adapter = kcalloc(1, sizeof(struct ixgbe_fwd_adapter), GFP_KERNEL); + if (!fwd_adapter) + return ERR_PTR(-ENOMEM); + + pool = find_first_zero_bit(&adapter->fwd_bitmask, 32); + adapter->num_rx_pools++; + set_bit(pool, &adapter->fwd_bitmask); + limit = find_last_bit(&adapter->fwd_bitmask, 32); + + /* Enable VMDq flag so device will be set in VM mode */ + adapter->flags |= IXGBE_FLAG_VMDQ_ENABLED | IXGBE_FLAG_SRIOV_ENABLED; + adapter->ring_feature[RING_F_VMDQ].limit = limit + 1; + adapter->ring_feature[RING_F_RSS].limit = vdev->num_tx_queues; + + /* Force reinit of ring allocation with VMDQ enabled */ + err = ixgbe_setup_tc(pdev, netdev_get_num_tc(pdev)); + if (err) + goto fwd_add_err; + fwd_adapter->pool = pool; + fwd_adapter->real_adapter = adapter; + err = ixgbe_fwd_ring_up(vdev, fwd_adapter); + if (err) + goto fwd_add_err; + netif_tx_start_all_queues(vdev); + return fwd_adapter; +fwd_add_err: + /* unwind counter and free adapter struct */ + netdev_info(pdev, + "%s: dfwd hardware acceleration failed\n", vdev->name); + clear_bit(pool, &adapter->fwd_bitmask); + adapter->num_rx_pools--; + kfree(fwd_adapter); + return ERR_PTR(err); +} + +static void ixgbe_fwd_del(struct net_device *pdev, void *priv) +{ + struct ixgbe_fwd_adapter *fwd_adapter = priv; + struct ixgbe_adapter *adapter = fwd_adapter->real_adapter; + unsigned int limit; + + clear_bit(fwd_adapter->pool, &adapter->fwd_bitmask); + adapter->num_rx_pools--; + + limit = find_last_bit(&adapter->fwd_bitmask, 32); + adapter->ring_feature[RING_F_VMDQ].limit = limit + 1; + ixgbe_fwd_ring_down(fwd_adapter->netdev, fwd_adapter); + ixgbe_setup_tc(pdev, netdev_get_num_tc(pdev)); + netdev_dbg(pdev, "pool %i:%i queues %i:%i VSI bitmask %lx\n", + fwd_adapter->pool, adapter->num_rx_pools, + fwd_adapter->rx_base_queue, + fwd_adapter->rx_base_queue + adapter->num_rx_queues_per_pool, + adapter->fwd_bitmask); + kfree(fwd_adapter); +} + +static netdev_tx_t ixgbe_fwd_xmit(struct sk_buff *skb, + struct net_device *dev, + void *priv) +{ + struct ixgbe_fwd_adapter *fwd_adapter = priv; + unsigned int queue; + struct ixgbe_ring *tx_ring; + + queue = skb->queue_mapping + fwd_adapter->tx_base_queue; + tx_ring = fwd_adapter->real_adapter->tx_ring[queue]; + + return __ixgbe_xmit_frame(skb, dev, tx_ring); +} + static const struct net_device_ops ixgbe_netdev_ops = { .ndo_open = ixgbe_open, .ndo_stop = ixgbe_close, @@ -7349,6 +7683,9 @@ static const struct net_device_ops ixgbe_netdev_ops = { .ndo_fdb_add = ixgbe_ndo_fdb_add, .ndo_bridge_setlink = ixgbe_ndo_bridge_setlink, .ndo_bridge_getlink = ixgbe_ndo_bridge_getlink, + .ndo_dfwd_add_station = ixgbe_fwd_add, + .ndo_dfwd_del_station = ixgbe_fwd_del, + .ndo_dfwd_start_xmit = ixgbe_fwd_xmit, }; /** @@ -7362,19 +7699,16 @@ static const struct net_device_ops ixgbe_netdev_ops = { **/ static inline int ixgbe_enumerate_functions(struct ixgbe_adapter *adapter) { - struct ixgbe_hw *hw = &adapter->hw; struct list_head *entry; int physfns = 0; - /* Some cards can not use the generic count PCIe functions method, and - * so must be hardcoded to the correct value. + /* Some cards can not use the generic count PCIe functions method, + * because they are behind a parent switch, so we hardcode these with + * the correct number of functions. */ - switch (hw->device_id) { - case IXGBE_DEV_ID_82599_SFP_SF_QP: - case IXGBE_DEV_ID_82599_QSFP_SF_QP: + if (ixgbe_pcie_from_parent(&adapter->hw)) { physfns = 4; - break; - default: + } else { list_for_each(entry, &adapter->pdev->bus_list) { struct pci_dev *pdev = list_entry(entry, struct pci_dev, bus_list); @@ -7653,7 +7987,8 @@ skip_sriov: NETIF_F_TSO | NETIF_F_TSO6 | NETIF_F_RXHASH | - NETIF_F_RXCSUM; + NETIF_F_RXCSUM | + NETIF_F_HW_L2FW_DOFFLOAD; netdev->hw_features = netdev->features; @@ -7759,29 +8094,6 @@ skip_sriov: if (ixgbe_pcie_from_parent(hw)) ixgbe_get_parent_bus_info(adapter); - /* print bus type/speed/width info */ - e_dev_info("(PCI Express:%s:%s) %pM\n", - (hw->bus.speed == ixgbe_bus_speed_8000 ? "8.0GT/s" : - hw->bus.speed == ixgbe_bus_speed_5000 ? "5.0GT/s" : - hw->bus.speed == ixgbe_bus_speed_2500 ? "2.5GT/s" : - "Unknown"), - (hw->bus.width == ixgbe_bus_width_pcie_x8 ? "Width x8" : - hw->bus.width == ixgbe_bus_width_pcie_x4 ? "Width x4" : - hw->bus.width == ixgbe_bus_width_pcie_x1 ? "Width x1" : - "Unknown"), - netdev->dev_addr); - - err = ixgbe_read_pba_string_generic(hw, part_str, IXGBE_PBANUM_LENGTH); - if (err) - strncpy(part_str, "Unknown", IXGBE_PBANUM_LENGTH); - if (ixgbe_is_sfp(hw) && hw->phy.sfp_type != ixgbe_sfp_type_not_present) - e_dev_info("MAC: %d, PHY: %d, SFP+: %d, PBA No: %s\n", - hw->mac.type, hw->phy.type, hw->phy.sfp_type, - part_str); - else - e_dev_info("MAC: %d, PHY: %d, PBA No: %s\n", - hw->mac.type, hw->phy.type, part_str); - /* calculate the expected PCIe bandwidth required for optimal * performance. Note that some older parts will never have enough * bandwidth due to being older generation PCIe parts. We clamp these @@ -7797,6 +8109,19 @@ skip_sriov: } ixgbe_check_minimum_link(adapter, expected_gts); + err = ixgbe_read_pba_string_generic(hw, part_str, IXGBE_PBANUM_LENGTH); + if (err) + strncpy(part_str, "Unknown", IXGBE_PBANUM_LENGTH); + if (ixgbe_is_sfp(hw) && hw->phy.sfp_type != ixgbe_sfp_type_not_present) + e_dev_info("MAC: %d, PHY: %d, SFP+: %d, PBA No: %s\n", + hw->mac.type, hw->phy.type, hw->phy.sfp_type, + part_str); + else + e_dev_info("MAC: %d, PHY: %d, PBA No: %s\n", + hw->mac.type, hw->phy.type, part_str); + + e_dev_info("%pM\n", netdev->dev_addr); + /* reset the hardware with the new settings */ err = hw->mac.ops.start_hw(hw); if (err == IXGBE_ERR_EEPROM_VERSION) { diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_phy.h b/drivers/net/ethernet/intel/ixgbe/ixgbe_phy.h index 24af12e..aae900a 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_phy.h +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_phy.h @@ -57,28 +57,28 @@ #define IXGBE_SFF_QSFP_DEVICE_TECH 0x93 /* Bitmasks */ -#define IXGBE_SFF_DA_PASSIVE_CABLE 0x4 -#define IXGBE_SFF_DA_ACTIVE_CABLE 0x8 -#define IXGBE_SFF_DA_SPEC_ACTIVE_LIMITING 0x4 -#define IXGBE_SFF_1GBASESX_CAPABLE 0x1 -#define IXGBE_SFF_1GBASELX_CAPABLE 0x2 -#define IXGBE_SFF_1GBASET_CAPABLE 0x8 -#define IXGBE_SFF_10GBASESR_CAPABLE 0x10 -#define IXGBE_SFF_10GBASELR_CAPABLE 0x20 -#define IXGBE_SFF_SOFT_RS_SELECT_MASK 0x8 -#define IXGBE_SFF_SOFT_RS_SELECT_10G 0x8 -#define IXGBE_SFF_SOFT_RS_SELECT_1G 0x0 -#define IXGBE_SFF_ADDRESSING_MODE 0x4 -#define IXGBE_SFF_QSFP_DA_ACTIVE_CABLE 0x1 -#define IXGBE_SFF_QSFP_DA_PASSIVE_CABLE 0x8 +#define IXGBE_SFF_DA_PASSIVE_CABLE 0x4 +#define IXGBE_SFF_DA_ACTIVE_CABLE 0x8 +#define IXGBE_SFF_DA_SPEC_ACTIVE_LIMITING 0x4 +#define IXGBE_SFF_1GBASESX_CAPABLE 0x1 +#define IXGBE_SFF_1GBASELX_CAPABLE 0x2 +#define IXGBE_SFF_1GBASET_CAPABLE 0x8 +#define IXGBE_SFF_10GBASESR_CAPABLE 0x10 +#define IXGBE_SFF_10GBASELR_CAPABLE 0x20 +#define IXGBE_SFF_SOFT_RS_SELECT_MASK 0x8 +#define IXGBE_SFF_SOFT_RS_SELECT_10G 0x8 +#define IXGBE_SFF_SOFT_RS_SELECT_1G 0x0 +#define IXGBE_SFF_ADDRESSING_MODE 0x4 +#define IXGBE_SFF_QSFP_DA_ACTIVE_CABLE 0x1 +#define IXGBE_SFF_QSFP_DA_PASSIVE_CABLE 0x8 #define IXGBE_SFF_QSFP_CONNECTOR_NOT_SEPARABLE 0x23 #define IXGBE_SFF_QSFP_TRANSMITER_850NM_VCSEL 0x0 -#define IXGBE_I2C_EEPROM_READ_MASK 0x100 -#define IXGBE_I2C_EEPROM_STATUS_MASK 0x3 -#define IXGBE_I2C_EEPROM_STATUS_NO_OPERATION 0x0 -#define IXGBE_I2C_EEPROM_STATUS_PASS 0x1 -#define IXGBE_I2C_EEPROM_STATUS_FAIL 0x2 -#define IXGBE_I2C_EEPROM_STATUS_IN_PROGRESS 0x3 +#define IXGBE_I2C_EEPROM_READ_MASK 0x100 +#define IXGBE_I2C_EEPROM_STATUS_MASK 0x3 +#define IXGBE_I2C_EEPROM_STATUS_NO_OPERATION 0x0 +#define IXGBE_I2C_EEPROM_STATUS_PASS 0x1 +#define IXGBE_I2C_EEPROM_STATUS_FAIL 0x2 +#define IXGBE_I2C_EEPROM_STATUS_IN_PROGRESS 0x3 /* Flow control defines */ #define IXGBE_TAF_SYM_PAUSE 0x400 diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c index 276d7b1..d6f0c0d 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c @@ -129,10 +129,6 @@ void ixgbe_enable_sriov(struct ixgbe_adapter *adapter) if (!pre_existing_vfs && !adapter->num_vfs) return; - if (!pre_existing_vfs) - dev_warn(&adapter->pdev->dev, - "Enabling SR-IOV VFs using the module parameter is deprecated - please use the pci sysfs interface.\n"); - /* If there are pre-existing VFs then we have to force * use of that many - over ride any module parameter value. * This may result from the user unloading the PF driver @@ -223,17 +219,19 @@ int ixgbe_disable_sriov(struct ixgbe_adapter *adapter) IXGBE_WRITE_FLUSH(hw); /* Disable VMDq flag so device will be set in VM mode */ - if (adapter->ring_feature[RING_F_VMDQ].limit == 1) + if (adapter->ring_feature[RING_F_VMDQ].limit == 1) { adapter->flags &= ~IXGBE_FLAG_VMDQ_ENABLED; - adapter->ring_feature[RING_F_VMDQ].offset = 0; + adapter->flags &= ~IXGBE_FLAG_SRIOV_ENABLED; + rss = min_t(int, IXGBE_MAX_RSS_INDICES, num_online_cpus()); + } else { + rss = min_t(int, IXGBE_MAX_L2A_QUEUES, num_online_cpus()); + } - rss = min_t(int, IXGBE_MAX_RSS_INDICES, num_online_cpus()); + adapter->ring_feature[RING_F_VMDQ].offset = 0; adapter->ring_feature[RING_F_RSS].limit = rss; /* take a breather then clean up driver data */ msleep(100); - - adapter->flags &= ~IXGBE_FLAG_SRIOV_ENABLED; return 0; } @@ -298,13 +296,10 @@ static int ixgbe_pci_sriov_disable(struct pci_dev *dev) err = ixgbe_disable_sriov(adapter); /* Only reinit if no error and state changed */ - if (!err && current_flags != adapter->flags) { - /* ixgbe_disable_sriov() doesn't clear VMDQ flag */ - adapter->flags &= ~IXGBE_FLAG_VMDQ_ENABLED; #ifdef CONFIG_PCI_IOV + if (!err && current_flags != adapter->flags) ixgbe_sriov_reinit(adapter); #endif - } return err; } @@ -558,7 +553,7 @@ static int ixgbe_set_vf_mac(struct ixgbe_adapter *adapter, struct ixgbe_hw *hw = &adapter->hw; int rar_entry = hw->mac.num_rar_entries - (vf + 1); - memcpy(adapter->vfinfo[vf].vf_mac_addresses, mac_addr, 6); + memcpy(adapter->vfinfo[vf].vf_mac_addresses, mac_addr, ETH_ALEN); hw->mac.ops.set_rar(hw, rar_entry, mac_addr, vf, IXGBE_RAH_AV); return 0; @@ -621,16 +616,13 @@ static int ixgbe_set_vf_macvlan(struct ixgbe_adapter *adapter, int ixgbe_vf_configuration(struct pci_dev *pdev, unsigned int event_mask) { - unsigned char vf_mac_addr[6]; struct ixgbe_adapter *adapter = pci_get_drvdata(pdev); unsigned int vfn = (event_mask & 0x3f); bool enable = ((event_mask & 0x10000000U) != 0); - if (enable) { - eth_zero_addr(vf_mac_addr); - memcpy(adapter->vfinfo[vfn].vf_mac_addresses, vf_mac_addr, 6); - } + if (enable) + eth_zero_addr(adapter->vfinfo[vfn].vf_mac_addresses); return 0; } diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h b/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h index 10775cb..7c19e96 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h @@ -561,6 +561,10 @@ struct ixgbe_thermal_sensor_data { #define IXGBE_RTTDQSEL 0x04904 #define IXGBE_RTTDT1C 0x04908 #define IXGBE_RTTDT1S 0x0490C +#define IXGBE_RTTQCNCR 0x08B00 +#define IXGBE_RTTQCNTG 0x04A90 +#define IXGBE_RTTBCNRD 0x0498C +#define IXGBE_RTTQCNRR 0x0498C #define IXGBE_RTTDTECC 0x04990 #define IXGBE_RTTDTECC_NO_BCN 0x00000100 #define IXGBE_RTTBCNRC 0x04984 @@ -570,6 +574,7 @@ struct ixgbe_thermal_sensor_data { #define IXGBE_RTTBCNRC_RF_INT_MASK \ (IXGBE_RTTBCNRC_RF_DEC_MASK << IXGBE_RTTBCNRC_RF_INT_SHIFT) #define IXGBE_RTTBCNRM 0x04980 +#define IXGBE_RTTQCNRM 0x04980 /* FCoE DMA Context Registers */ #define IXGBE_FCPTRL 0x02410 /* FC User Desc. PTR Low */ diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c index 389324f..24b80a6 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c @@ -32,12 +32,12 @@ #include "ixgbe.h" #include "ixgbe_phy.h" -#define IXGBE_X540_MAX_TX_QUEUES 128 -#define IXGBE_X540_MAX_RX_QUEUES 128 -#define IXGBE_X540_RAR_ENTRIES 128 -#define IXGBE_X540_MC_TBL_SIZE 128 -#define IXGBE_X540_VFT_TBL_SIZE 128 -#define IXGBE_X540_RX_PB_SIZE 384 +#define IXGBE_X540_MAX_TX_QUEUES 128 +#define IXGBE_X540_MAX_RX_QUEUES 128 +#define IXGBE_X540_RAR_ENTRIES 128 +#define IXGBE_X540_MC_TBL_SIZE 128 +#define IXGBE_X540_VFT_TBL_SIZE 128 +#define IXGBE_X540_RX_PB_SIZE 384 static s32 ixgbe_update_flash_X540(struct ixgbe_hw *hw); static s32 ixgbe_poll_flash_update_done_X540(struct ixgbe_hw *hw); diff --git a/drivers/net/ethernet/intel/ixgbevf/ethtool.c b/drivers/net/ethernet/intel/ixgbevf/ethtool.c index c9d0c12..54d9ace 100644 --- a/drivers/net/ethernet/intel/ixgbevf/ethtool.c +++ b/drivers/net/ethernet/intel/ixgbevf/ethtool.c @@ -45,16 +45,27 @@ struct ixgbe_stats { char stat_string[ETH_GSTRING_LEN]; - int sizeof_stat; - int stat_offset; - int base_stat_offset; - int saved_reset_offset; + struct { + int sizeof_stat; + int stat_offset; + int base_stat_offset; + int saved_reset_offset; + }; }; -#define IXGBEVF_STAT(m, b, r) sizeof(((struct ixgbevf_adapter *)0)->m), \ - offsetof(struct ixgbevf_adapter, m), \ - offsetof(struct ixgbevf_adapter, b), \ - offsetof(struct ixgbevf_adapter, r) +#define IXGBEVF_STAT(m, b, r) { \ + .sizeof_stat = FIELD_SIZEOF(struct ixgbevf_adapter, m), \ + .stat_offset = offsetof(struct ixgbevf_adapter, m), \ + .base_stat_offset = offsetof(struct ixgbevf_adapter, b), \ + .saved_reset_offset = offsetof(struct ixgbevf_adapter, r) \ +} + +#define IXGBEVF_ZSTAT(m) { \ + .sizeof_stat = FIELD_SIZEOF(struct ixgbevf_adapter, m), \ + .stat_offset = offsetof(struct ixgbevf_adapter, m), \ + .base_stat_offset = -1, \ + .saved_reset_offset = -1 \ +} static const struct ixgbe_stats ixgbe_gstrings_stats[] = { {"rx_packets", IXGBEVF_STAT(stats.vfgprc, stats.base_vfgprc, @@ -65,15 +76,20 @@ static const struct ixgbe_stats ixgbe_gstrings_stats[] = { stats.saved_reset_vfgorc)}, {"tx_bytes", IXGBEVF_STAT(stats.vfgotc, stats.base_vfgotc, stats.saved_reset_vfgotc)}, - {"tx_busy", IXGBEVF_STAT(tx_busy, zero_base, zero_base)}, + {"tx_busy", IXGBEVF_ZSTAT(tx_busy)}, {"multicast", IXGBEVF_STAT(stats.vfmprc, stats.base_vfmprc, stats.saved_reset_vfmprc)}, - {"rx_csum_offload_good", IXGBEVF_STAT(hw_csum_rx_good, zero_base, - zero_base)}, - {"rx_csum_offload_errors", IXGBEVF_STAT(hw_csum_rx_error, zero_base, - zero_base)}, - {"tx_csum_offload_ctxt", IXGBEVF_STAT(hw_csum_tx_good, zero_base, - zero_base)}, + {"rx_csum_offload_good", IXGBEVF_ZSTAT(hw_csum_rx_good)}, + {"rx_csum_offload_errors", IXGBEVF_ZSTAT(hw_csum_rx_error)}, + {"tx_csum_offload_ctxt", IXGBEVF_ZSTAT(hw_csum_tx_good)}, +#ifdef BP_EXTENDED_STATS + {"rx_bp_poll_yield", IXGBEVF_ZSTAT(bp_rx_yields)}, + {"rx_bp_cleaned", IXGBEVF_ZSTAT(bp_rx_cleaned)}, + {"rx_bp_misses", IXGBEVF_ZSTAT(bp_rx_missed)}, + {"tx_bp_napi_yield", IXGBEVF_ZSTAT(bp_tx_yields)}, + {"tx_bp_cleaned", IXGBEVF_ZSTAT(bp_tx_cleaned)}, + {"tx_bp_misses", IXGBEVF_ZSTAT(bp_tx_missed)}, +#endif }; #define IXGBE_QUEUE_STATS_LEN 0 @@ -140,58 +156,10 @@ static void ixgbevf_set_msglevel(struct net_device *netdev, u32 data) #define IXGBE_GET_STAT(_A_, _R_) (_A_->stats._R_) -static char *ixgbevf_reg_names[] = { - "IXGBE_VFCTRL", - "IXGBE_VFSTATUS", - "IXGBE_VFLINKS", - "IXGBE_VFRXMEMWRAP", - "IXGBE_VFFRTIMER", - "IXGBE_VTEICR", - "IXGBE_VTEICS", - "IXGBE_VTEIMS", - "IXGBE_VTEIMC", - "IXGBE_VTEIAC", - "IXGBE_VTEIAM", - "IXGBE_VTEITR", - "IXGBE_VTIVAR", - "IXGBE_VTIVAR_MISC", - "IXGBE_VFRDBAL0", - "IXGBE_VFRDBAL1", - "IXGBE_VFRDBAH0", - "IXGBE_VFRDBAH1", - "IXGBE_VFRDLEN0", - "IXGBE_VFRDLEN1", - "IXGBE_VFRDH0", - "IXGBE_VFRDH1", - "IXGBE_VFRDT0", - "IXGBE_VFRDT1", - "IXGBE_VFRXDCTL0", - "IXGBE_VFRXDCTL1", - "IXGBE_VFSRRCTL0", - "IXGBE_VFSRRCTL1", - "IXGBE_VFPSRTYPE", - "IXGBE_VFTDBAL0", - "IXGBE_VFTDBAL1", - "IXGBE_VFTDBAH0", - "IXGBE_VFTDBAH1", - "IXGBE_VFTDLEN0", - "IXGBE_VFTDLEN1", - "IXGBE_VFTDH0", - "IXGBE_VFTDH1", - "IXGBE_VFTDT0", - "IXGBE_VFTDT1", - "IXGBE_VFTXDCTL0", - "IXGBE_VFTXDCTL1", - "IXGBE_VFTDWBAL0", - "IXGBE_VFTDWBAL1", - "IXGBE_VFTDWBAH0", - "IXGBE_VFTDWBAH1" -}; - - static int ixgbevf_get_regs_len(struct net_device *netdev) { - return (ARRAY_SIZE(ixgbevf_reg_names)) * sizeof(u32); +#define IXGBE_REGS_LEN 45 + return IXGBE_REGS_LEN * sizeof(u32); } static void ixgbevf_get_regs(struct net_device *netdev, @@ -264,9 +232,6 @@ static void ixgbevf_get_regs(struct net_device *netdev, regs_buff[41 + i] = IXGBE_READ_REG(hw, IXGBE_VFTDWBAL(i)); for (i = 0; i < 2; i++) regs_buff[43 + i] = IXGBE_READ_REG(hw, IXGBE_VFTDWBAH(i)); - - for (i = 0; i < ARRAY_SIZE(ixgbevf_reg_names); i++) - hw_dbg(hw, "%s\t%8.8x\n", ixgbevf_reg_names[i], regs_buff[i]); } static void ixgbevf_get_drvinfo(struct net_device *netdev, @@ -441,22 +406,50 @@ static void ixgbevf_get_ethtool_stats(struct net_device *netdev, struct ethtool_stats *stats, u64 *data) { struct ixgbevf_adapter *adapter = netdev_priv(netdev); + char *base = (char *) adapter; int i; +#ifdef BP_EXTENDED_STATS + u64 rx_yields = 0, rx_cleaned = 0, rx_missed = 0, + tx_yields = 0, tx_cleaned = 0, tx_missed = 0; + + for (i = 0; i < adapter->num_rx_queues; i++) { + rx_yields += adapter->rx_ring[i].bp_yields; + rx_cleaned += adapter->rx_ring[i].bp_cleaned; + rx_yields += adapter->rx_ring[i].bp_yields; + } + + for (i = 0; i < adapter->num_tx_queues; i++) { + tx_yields += adapter->tx_ring[i].bp_yields; + tx_cleaned += adapter->tx_ring[i].bp_cleaned; + tx_yields += adapter->tx_ring[i].bp_yields; + } + + adapter->bp_rx_yields = rx_yields; + adapter->bp_rx_cleaned = rx_cleaned; + adapter->bp_rx_missed = rx_missed; + + adapter->bp_tx_yields = tx_yields; + adapter->bp_tx_cleaned = tx_cleaned; + adapter->bp_tx_missed = tx_missed; +#endif ixgbevf_update_stats(adapter); for (i = 0; i < IXGBE_GLOBAL_STATS_LEN; i++) { - char *p = (char *)adapter + - ixgbe_gstrings_stats[i].stat_offset; - char *b = (char *)adapter + - ixgbe_gstrings_stats[i].base_stat_offset; - char *r = (char *)adapter + - ixgbe_gstrings_stats[i].saved_reset_offset; - data[i] = ((ixgbe_gstrings_stats[i].sizeof_stat == - sizeof(u64)) ? *(u64 *)p : *(u32 *)p) - - ((ixgbe_gstrings_stats[i].sizeof_stat == - sizeof(u64)) ? *(u64 *)b : *(u32 *)b) + - ((ixgbe_gstrings_stats[i].sizeof_stat == - sizeof(u64)) ? *(u64 *)r : *(u32 *)r); + char *p = base + ixgbe_gstrings_stats[i].stat_offset; + char *b = base + ixgbe_gstrings_stats[i].base_stat_offset; + char *r = base + ixgbe_gstrings_stats[i].saved_reset_offset; + + if (ixgbe_gstrings_stats[i].sizeof_stat == sizeof(u64)) { + if (ixgbe_gstrings_stats[i].base_stat_offset >= 0) + data[i] = *(u64 *)p - *(u64 *)b + *(u64 *)r; + else + data[i] = *(u64 *)p; + } else { + if (ixgbe_gstrings_stats[i].base_stat_offset >= 0) + data[i] = *(u32 *)p - *(u32 *)b + *(u32 *)r; + else + data[i] = *(u32 *)p; + } } } @@ -685,6 +678,85 @@ static int ixgbevf_nway_reset(struct net_device *netdev) return 0; } +static int ixgbevf_get_coalesce(struct net_device *netdev, + struct ethtool_coalesce *ec) +{ + struct ixgbevf_adapter *adapter = netdev_priv(netdev); + + /* only valid if in constant ITR mode */ + if (adapter->rx_itr_setting <= 1) + ec->rx_coalesce_usecs = adapter->rx_itr_setting; + else + ec->rx_coalesce_usecs = adapter->rx_itr_setting >> 2; + + /* if in mixed tx/rx queues per vector mode, report only rx settings */ + if (adapter->q_vector[0]->tx.count && adapter->q_vector[0]->rx.count) + return 0; + + /* only valid if in constant ITR mode */ + if (adapter->tx_itr_setting <= 1) + ec->tx_coalesce_usecs = adapter->tx_itr_setting; + else + ec->tx_coalesce_usecs = adapter->tx_itr_setting >> 2; + + return 0; +} + +static int ixgbevf_set_coalesce(struct net_device *netdev, + struct ethtool_coalesce *ec) +{ + struct ixgbevf_adapter *adapter = netdev_priv(netdev); + struct ixgbevf_q_vector *q_vector; + int num_vectors, i; + u16 tx_itr_param, rx_itr_param; + + /* don't accept tx specific changes if we've got mixed RxTx vectors */ + if (adapter->q_vector[0]->tx.count && adapter->q_vector[0]->rx.count + && ec->tx_coalesce_usecs) + return -EINVAL; + + + if ((ec->rx_coalesce_usecs > (IXGBE_MAX_EITR >> 2)) || + (ec->tx_coalesce_usecs > (IXGBE_MAX_EITR >> 2))) + return -EINVAL; + + if (ec->rx_coalesce_usecs > 1) + adapter->rx_itr_setting = ec->rx_coalesce_usecs << 2; + else + adapter->rx_itr_setting = ec->rx_coalesce_usecs; + + if (adapter->rx_itr_setting == 1) + rx_itr_param = IXGBE_20K_ITR; + else + rx_itr_param = adapter->rx_itr_setting; + + + if (ec->tx_coalesce_usecs > 1) + adapter->tx_itr_setting = ec->tx_coalesce_usecs << 2; + else + adapter->tx_itr_setting = ec->tx_coalesce_usecs; + + if (adapter->tx_itr_setting == 1) + tx_itr_param = IXGBE_10K_ITR; + else + tx_itr_param = adapter->tx_itr_setting; + + num_vectors = adapter->num_msix_vectors - NON_Q_VECTORS; + + for (i = 0; i < num_vectors; i++) { + q_vector = adapter->q_vector[i]; + if (q_vector->tx.count && !q_vector->rx.count) + /* tx only */ + q_vector->itr = tx_itr_param; + else + /* rx only or mixed */ + q_vector->itr = rx_itr_param; + ixgbevf_write_eitr(q_vector); + } + + return 0; +} + static const struct ethtool_ops ixgbevf_ethtool_ops = { .get_settings = ixgbevf_get_settings, .get_drvinfo = ixgbevf_get_drvinfo, @@ -700,6 +772,8 @@ static const struct ethtool_ops ixgbevf_ethtool_ops = { .get_sset_count = ixgbevf_get_sset_count, .get_strings = ixgbevf_get_strings, .get_ethtool_stats = ixgbevf_get_ethtool_stats, + .get_coalesce = ixgbevf_get_coalesce, + .set_coalesce = ixgbevf_set_coalesce, }; void ixgbevf_set_ethtool_ops(struct net_device *netdev) diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf.h b/drivers/net/ethernet/intel/ixgbevf/ixgbevf.h index fff0d98..8971e2d 100644 --- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf.h +++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf.h @@ -38,6 +38,11 @@ #include "vf.h" +#ifdef CONFIG_NET_RX_BUSY_POLL +#include <net/busy_poll.h> +#define BP_EXTENDED_STATS +#endif + /* wrapper around a pointer to a socket buffer, * so a DMA handle can be stored along with the buffer */ struct ixgbevf_tx_buffer { @@ -76,6 +81,11 @@ struct ixgbevf_ring { struct u64_stats_sync syncp; u64 hw_csum_rx_error; u64 hw_csum_rx_good; +#ifdef BP_EXTENDED_STATS + u64 bp_yields; + u64 bp_misses; + u64 bp_cleaned; +#endif u16 head; u16 tail; @@ -145,7 +155,118 @@ struct ixgbevf_q_vector { struct napi_struct napi; struct ixgbevf_ring_container rx, tx; char name[IFNAMSIZ + 9]; +#ifdef CONFIG_NET_RX_BUSY_POLL + unsigned int state; +#define IXGBEVF_QV_STATE_IDLE 0 +#define IXGBEVF_QV_STATE_NAPI 1 /* NAPI owns this QV */ +#define IXGBEVF_QV_STATE_POLL 2 /* poll owns this QV */ +#define IXGBEVF_QV_STATE_DISABLED 4 /* QV is disabled */ +#define IXGBEVF_QV_OWNED (IXGBEVF_QV_STATE_NAPI | IXGBEVF_QV_STATE_POLL) +#define IXGBEVF_QV_LOCKED (IXGBEVF_QV_OWNED | IXGBEVF_QV_STATE_DISABLED) +#define IXGBEVF_QV_STATE_NAPI_YIELD 8 /* NAPI yielded this QV */ +#define IXGBEVF_QV_STATE_POLL_YIELD 16 /* poll yielded this QV */ +#define IXGBEVF_QV_YIELD (IXGBEVF_QV_STATE_NAPI_YIELD | IXGBEVF_QV_STATE_POLL_YIELD) +#define IXGBEVF_QV_USER_PEND (IXGBEVF_QV_STATE_POLL | IXGBEVF_QV_STATE_POLL_YIELD) + spinlock_t lock; +#endif /* CONFIG_NET_RX_BUSY_POLL */ }; +#ifdef CONFIG_NET_RX_BUSY_POLL +static inline void ixgbevf_qv_init_lock(struct ixgbevf_q_vector *q_vector) +{ + + spin_lock_init(&q_vector->lock); + q_vector->state = IXGBEVF_QV_STATE_IDLE; +} + +/* called from the device poll routine to get ownership of a q_vector */ +static inline bool ixgbevf_qv_lock_napi(struct ixgbevf_q_vector *q_vector) +{ + int rc = true; + spin_lock_bh(&q_vector->lock); + if (q_vector->state & IXGBEVF_QV_LOCKED) { + WARN_ON(q_vector->state & IXGBEVF_QV_STATE_NAPI); + q_vector->state |= IXGBEVF_QV_STATE_NAPI_YIELD; + rc = false; +#ifdef BP_EXTENDED_STATS + q_vector->tx.ring->bp_yields++; +#endif + } else { + /* we don't care if someone yielded */ + q_vector->state = IXGBEVF_QV_STATE_NAPI; + } + spin_unlock_bh(&q_vector->lock); + return rc; +} + +/* returns true is someone tried to get the qv while napi had it */ +static inline bool ixgbevf_qv_unlock_napi(struct ixgbevf_q_vector *q_vector) +{ + int rc = false; + spin_lock_bh(&q_vector->lock); + WARN_ON(q_vector->state & (IXGBEVF_QV_STATE_POLL | + IXGBEVF_QV_STATE_NAPI_YIELD)); + + if (q_vector->state & IXGBEVF_QV_STATE_POLL_YIELD) + rc = true; + /* reset state to idle, unless QV is disabled */ + q_vector->state &= IXGBEVF_QV_STATE_DISABLED; + spin_unlock_bh(&q_vector->lock); + return rc; +} + +/* called from ixgbevf_low_latency_poll() */ +static inline bool ixgbevf_qv_lock_poll(struct ixgbevf_q_vector *q_vector) +{ + int rc = true; + spin_lock_bh(&q_vector->lock); + if ((q_vector->state & IXGBEVF_QV_LOCKED)) { + q_vector->state |= IXGBEVF_QV_STATE_POLL_YIELD; + rc = false; +#ifdef BP_EXTENDED_STATS + q_vector->rx.ring->bp_yields++; +#endif + } else { + /* preserve yield marks */ + q_vector->state |= IXGBEVF_QV_STATE_POLL; + } + spin_unlock_bh(&q_vector->lock); + return rc; +} + +/* returns true if someone tried to get the qv while it was locked */ +static inline bool ixgbevf_qv_unlock_poll(struct ixgbevf_q_vector *q_vector) +{ + int rc = false; + spin_lock_bh(&q_vector->lock); + WARN_ON(q_vector->state & (IXGBEVF_QV_STATE_NAPI)); + + if (q_vector->state & IXGBEVF_QV_STATE_POLL_YIELD) + rc = true; + /* reset state to idle, unless QV is disabled */ + q_vector->state &= IXGBEVF_QV_STATE_DISABLED; + spin_unlock_bh(&q_vector->lock); + return rc; +} + +/* true if a socket is polling, even if it did not get the lock */ +static inline bool ixgbevf_qv_busy_polling(struct ixgbevf_q_vector *q_vector) +{ + WARN_ON(!(q_vector->state & IXGBEVF_QV_OWNED)); + return q_vector->state & IXGBEVF_QV_USER_PEND; +} + +/* false if QV is currently owned */ +static inline bool ixgbevf_qv_disable(struct ixgbevf_q_vector *q_vector) +{ + int rc = true; + spin_lock_bh(&q_vector->lock); + if (q_vector->state & IXGBEVF_QV_OWNED) + rc = false; + spin_unlock_bh(&q_vector->lock); + return rc; +} + +#endif /* CONFIG_NET_RX_BUSY_POLL */ /* * microsecond values for various ITR rates shifted by 2 to fit itr register @@ -165,9 +286,13 @@ struct ixgbevf_q_vector { ((_eitr) ? (1000000000 / ((_eitr) * 256)) : 8) #define EITR_REG_TO_INTS_PER_SEC EITR_INTS_PER_SEC_TO_REG -#define IXGBE_DESC_UNUSED(R) \ - ((((R)->next_to_clean > (R)->next_to_use) ? 0 : (R)->count) + \ - (R)->next_to_clean - (R)->next_to_use - 1) +static inline u16 ixgbevf_desc_unused(struct ixgbevf_ring *ring) +{ + u16 ntc = ring->next_to_clean; + u16 ntu = ring->next_to_use; + + return ((ntc > ntu) ? 0 : ring->count) + ntc - ntu - 1; +} #define IXGBEVF_RX_DESC(R, i) \ (&(((union ixgbe_adv_rx_desc *)((R)->desc))[i])) @@ -240,7 +365,6 @@ struct ixgbevf_adapter { struct ixgbe_hw hw; u16 msg_enable; struct ixgbevf_hw_stats stats; - u64 zero_base; /* Interrupt Throttle Rate */ u32 eitr_param; @@ -249,6 +373,16 @@ struct ixgbevf_adapter { unsigned int tx_ring_count; unsigned int rx_ring_count; +#ifdef BP_EXTENDED_STATS + u64 bp_rx_yields; + u64 bp_rx_cleaned; + u64 bp_rx_missed; + + u64 bp_tx_yields; + u64 bp_tx_cleaned; + u64 bp_tx_missed; +#endif + u32 link_speed; bool link_up; @@ -281,27 +415,25 @@ extern const struct ixgbe_mbx_operations ixgbevf_mbx_ops; extern const char ixgbevf_driver_name[]; extern const char ixgbevf_driver_version[]; -extern void ixgbevf_up(struct ixgbevf_adapter *adapter); -extern void ixgbevf_down(struct ixgbevf_adapter *adapter); -extern void ixgbevf_reinit_locked(struct ixgbevf_adapter *adapter); -extern void ixgbevf_reset(struct ixgbevf_adapter *adapter); -extern void ixgbevf_set_ethtool_ops(struct net_device *netdev); -extern int ixgbevf_setup_rx_resources(struct ixgbevf_adapter *, - struct ixgbevf_ring *); -extern int ixgbevf_setup_tx_resources(struct ixgbevf_adapter *, - struct ixgbevf_ring *); -extern void ixgbevf_free_rx_resources(struct ixgbevf_adapter *, - struct ixgbevf_ring *); -extern void ixgbevf_free_tx_resources(struct ixgbevf_adapter *, - struct ixgbevf_ring *); -extern void ixgbevf_update_stats(struct ixgbevf_adapter *adapter); -extern int ethtool_ioctl(struct ifreq *ifr); - -extern void ixgbe_napi_add_all(struct ixgbevf_adapter *adapter); -extern void ixgbe_napi_del_all(struct ixgbevf_adapter *adapter); +void ixgbevf_up(struct ixgbevf_adapter *adapter); +void ixgbevf_down(struct ixgbevf_adapter *adapter); +void ixgbevf_reinit_locked(struct ixgbevf_adapter *adapter); +void ixgbevf_reset(struct ixgbevf_adapter *adapter); +void ixgbevf_set_ethtool_ops(struct net_device *netdev); +int ixgbevf_setup_rx_resources(struct ixgbevf_adapter *, struct ixgbevf_ring *); +int ixgbevf_setup_tx_resources(struct ixgbevf_adapter *, struct ixgbevf_ring *); +void ixgbevf_free_rx_resources(struct ixgbevf_adapter *, struct ixgbevf_ring *); +void ixgbevf_free_tx_resources(struct ixgbevf_adapter *, struct ixgbevf_ring *); +void ixgbevf_update_stats(struct ixgbevf_adapter *adapter); +int ethtool_ioctl(struct ifreq *ifr); + +extern void ixgbevf_write_eitr(struct ixgbevf_q_vector *q_vector); + +void ixgbe_napi_add_all(struct ixgbevf_adapter *adapter); +void ixgbe_napi_del_all(struct ixgbevf_adapter *adapter); #ifdef DEBUG -extern char *ixgbevf_get_hw_dev_name(struct ixgbe_hw *hw); +char *ixgbevf_get_hw_dev_name(struct ixgbe_hw *hw); #define hw_dbg(hw, format, arg...) \ printk(KERN_DEBUG "%s: " format, ixgbevf_get_hw_dev_name(hw), ##arg) #else diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c index 59a62bb..038bfc8 100644 --- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c @@ -58,7 +58,7 @@ const char ixgbevf_driver_name[] = "ixgbevf"; static const char ixgbevf_driver_string[] = "Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver"; -#define DRV_VERSION "2.7.12-k" +#define DRV_VERSION "2.11.3-k" const char ixgbevf_driver_version[] = DRV_VERSION; static char ixgbevf_copyright[] = "Copyright (c) 2009 - 2012 Intel Corporation."; @@ -251,7 +251,7 @@ static bool ixgbevf_clean_tx_irq(struct ixgbevf_q_vector *q_vector, #define TX_WAKE_THRESHOLD (DESC_NEEDED * 2) if (unlikely(count && netif_carrier_ok(tx_ring->netdev) && - (IXGBE_DESC_UNUSED(tx_ring) >= TX_WAKE_THRESHOLD))) { + (ixgbevf_desc_unused(tx_ring) >= TX_WAKE_THRESHOLD))) { /* Make sure that anybody stopping the queue after this * sees the new next_to_clean. */ @@ -300,6 +300,30 @@ static void ixgbevf_receive_skb(struct ixgbevf_q_vector *q_vector, } /** + * ixgbevf_rx_skb - Helper function to determine proper Rx method + * @q_vector: structure containing interrupt and ring information + * @skb: packet to send up + * @status: hardware indication of status of receive + * @rx_desc: rx descriptor + **/ +static void ixgbevf_rx_skb(struct ixgbevf_q_vector *q_vector, + struct sk_buff *skb, u8 status, + union ixgbe_adv_rx_desc *rx_desc) +{ +#ifdef CONFIG_NET_RX_BUSY_POLL + skb_mark_napi_id(skb, &q_vector->napi); + + if (ixgbevf_qv_busy_polling(q_vector)) { + netif_receive_skb(skb); + /* exit early if we busy polled */ + return; + } +#endif /* CONFIG_NET_RX_BUSY_POLL */ + + ixgbevf_receive_skb(q_vector, skb, status, rx_desc); +} + +/** * ixgbevf_rx_checksum - indicate in skb if hw indicated a good cksum * @ring: pointer to Rx descriptor ring structure * @status_err: hardware indication of status of receive @@ -396,9 +420,9 @@ static inline void ixgbevf_irq_enable_queues(struct ixgbevf_adapter *adapter, IXGBE_WRITE_REG(hw, IXGBE_VTEIMS, qmask); } -static bool ixgbevf_clean_rx_irq(struct ixgbevf_q_vector *q_vector, - struct ixgbevf_ring *rx_ring, - int budget) +static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vector *q_vector, + struct ixgbevf_ring *rx_ring, + int budget) { struct ixgbevf_adapter *adapter = q_vector->adapter; struct pci_dev *pdev = adapter->pdev; @@ -473,15 +497,6 @@ static bool ixgbevf_clean_rx_irq(struct ixgbevf_q_vector *q_vector, total_rx_bytes += skb->len; total_rx_packets++; - /* - * Work around issue of some types of VM to VM loop back - * packets not getting split correctly - */ - if (staterr & IXGBE_RXD_STAT_LB) { - u32 header_fixup_len = skb_headlen(skb); - if (header_fixup_len < 14) - skb_push(skb, header_fixup_len); - } skb->protocol = eth_type_trans(skb, rx_ring->netdev); /* Workaround hardware that can't do proper VEPA multicast @@ -494,7 +509,7 @@ static bool ixgbevf_clean_rx_irq(struct ixgbevf_q_vector *q_vector, goto next_desc; } - ixgbevf_receive_skb(q_vector, skb, staterr, rx_desc); + ixgbevf_rx_skb(q_vector, skb, staterr, rx_desc); next_desc: rx_desc->wb.upper.status_error = 0; @@ -514,7 +529,7 @@ next_desc: } rx_ring->next_to_clean = i; - cleaned_count = IXGBE_DESC_UNUSED(rx_ring); + cleaned_count = ixgbevf_desc_unused(rx_ring); if (cleaned_count) ixgbevf_alloc_rx_buffers(adapter, rx_ring, cleaned_count); @@ -526,7 +541,7 @@ next_desc: q_vector->rx.total_packets += total_rx_packets; q_vector->rx.total_bytes += total_rx_bytes; - return !!budget; + return total_rx_packets; } /** @@ -549,6 +564,11 @@ static int ixgbevf_poll(struct napi_struct *napi, int budget) ixgbevf_for_each_ring(ring, q_vector->tx) clean_complete &= ixgbevf_clean_tx_irq(q_vector, ring); +#ifdef CONFIG_NET_RX_BUSY_POLL + if (!ixgbevf_qv_lock_napi(q_vector)) + return budget; +#endif + /* attempt to distribute budget to each queue fairly, but don't allow * the budget to go below 1 because we'll exit polling */ if (q_vector->rx.count > 1) @@ -558,10 +578,15 @@ static int ixgbevf_poll(struct napi_struct *napi, int budget) adapter->flags |= IXGBE_FLAG_IN_NETPOLL; ixgbevf_for_each_ring(ring, q_vector->rx) - clean_complete &= ixgbevf_clean_rx_irq(q_vector, ring, - per_ring_budget); + clean_complete &= (ixgbevf_clean_rx_irq(q_vector, ring, + per_ring_budget) + < per_ring_budget); adapter->flags &= ~IXGBE_FLAG_IN_NETPOLL; +#ifdef CONFIG_NET_RX_BUSY_POLL + ixgbevf_qv_unlock_napi(q_vector); +#endif + /* If all work not completed, return budget and keep polling */ if (!clean_complete) return budget; @@ -580,7 +605,7 @@ static int ixgbevf_poll(struct napi_struct *napi, int budget) * ixgbevf_write_eitr - write VTEITR register in hardware specific way * @q_vector: structure containing interrupt and ring information */ -static void ixgbevf_write_eitr(struct ixgbevf_q_vector *q_vector) +void ixgbevf_write_eitr(struct ixgbevf_q_vector *q_vector) { struct ixgbevf_adapter *adapter = q_vector->adapter; struct ixgbe_hw *hw = &adapter->hw; @@ -596,6 +621,40 @@ static void ixgbevf_write_eitr(struct ixgbevf_q_vector *q_vector) IXGBE_WRITE_REG(hw, IXGBE_VTEITR(v_idx), itr_reg); } +#ifdef CONFIG_NET_RX_BUSY_POLL +/* must be called with local_bh_disable()d */ +static int ixgbevf_busy_poll_recv(struct napi_struct *napi) +{ + struct ixgbevf_q_vector *q_vector = + container_of(napi, struct ixgbevf_q_vector, napi); + struct ixgbevf_adapter *adapter = q_vector->adapter; + struct ixgbevf_ring *ring; + int found = 0; + + if (test_bit(__IXGBEVF_DOWN, &adapter->state)) + return LL_FLUSH_FAILED; + + if (!ixgbevf_qv_lock_poll(q_vector)) + return LL_FLUSH_BUSY; + + ixgbevf_for_each_ring(ring, q_vector->rx) { + found = ixgbevf_clean_rx_irq(q_vector, ring, 4); +#ifdef BP_EXTENDED_STATS + if (found) + ring->bp_cleaned += found; + else + ring->bp_misses++; +#endif + if (found) + break; + } + + ixgbevf_qv_unlock_poll(q_vector); + + return found; +} +#endif /* CONFIG_NET_RX_BUSY_POLL */ + /** * ixgbevf_configure_msix - Configure MSI-X hardware * @adapter: board private structure @@ -756,37 +815,12 @@ static void ixgbevf_set_itr(struct ixgbevf_q_vector *q_vector) static irqreturn_t ixgbevf_msix_other(int irq, void *data) { struct ixgbevf_adapter *adapter = data; - struct pci_dev *pdev = adapter->pdev; struct ixgbe_hw *hw = &adapter->hw; - u32 msg; - bool got_ack = false; hw->mac.get_link_status = 1; - if (!hw->mbx.ops.check_for_ack(hw)) - got_ack = true; - - if (!hw->mbx.ops.check_for_msg(hw)) { - hw->mbx.ops.read(hw, &msg, 1); - - if ((msg & IXGBE_MBVFICR_VFREQ_MASK) == IXGBE_PF_CONTROL_MSG) { - mod_timer(&adapter->watchdog_timer, - round_jiffies(jiffies + 1)); - adapter->link_up = false; - } - if (msg & IXGBE_VT_MSGTYPE_NACK) - dev_info(&pdev->dev, - "Last Request of type %2.2x to PF Nacked\n", - msg & 0xFF); - hw->mbx.v2p_mailbox |= IXGBE_VFMAILBOX_PFSTS; - } - - /* checking for the ack clears the PFACK bit. Place - * it back in the v2p_mailbox cache so that anyone - * polling for an ack will not miss it - */ - if (got_ack) - hw->mbx.v2p_mailbox |= IXGBE_VFMAILBOX_PFACK; + if (!test_bit(__IXGBEVF_DOWN, &adapter->state)) + mod_timer(&adapter->watchdog_timer, jiffies); IXGBE_WRITE_REG(hw, IXGBE_VTEIMS, adapter->eims_other); @@ -1107,6 +1141,21 @@ static void ixgbevf_configure_srrctl(struct ixgbevf_adapter *adapter, int index) IXGBE_WRITE_REG(hw, IXGBE_VFSRRCTL(index), srrctl); } +static void ixgbevf_setup_psrtype(struct ixgbevf_adapter *adapter) +{ + struct ixgbe_hw *hw = &adapter->hw; + + /* PSRTYPE must be initialized in 82599 */ + u32 psrtype = IXGBE_PSRTYPE_TCPHDR | IXGBE_PSRTYPE_UDPHDR | + IXGBE_PSRTYPE_IPV4HDR | IXGBE_PSRTYPE_IPV6HDR | + IXGBE_PSRTYPE_L2HDR; + + if (adapter->num_rx_queues > 1) + psrtype |= 1 << 29; + + IXGBE_WRITE_REG(hw, IXGBE_VFPSRTYPE, psrtype); +} + static void ixgbevf_set_rx_buffer_len(struct ixgbevf_adapter *adapter) { struct ixgbe_hw *hw = &adapter->hw; @@ -1154,8 +1203,7 @@ static void ixgbevf_configure_rx(struct ixgbevf_adapter *adapter) int i, j; u32 rdlen; - /* PSRTYPE must be initialized in 82599 */ - IXGBE_WRITE_REG(hw, IXGBE_VFPSRTYPE, 0); + ixgbevf_setup_psrtype(adapter); /* set_rx_buffer_len must be called before ring initialization */ ixgbevf_set_rx_buffer_len(adapter); @@ -1293,6 +1341,9 @@ static void ixgbevf_napi_enable_all(struct ixgbevf_adapter *adapter) for (q_idx = 0; q_idx < q_vectors; q_idx++) { q_vector = adapter->q_vector[q_idx]; +#ifdef CONFIG_NET_RX_BUSY_POLL + ixgbevf_qv_init_lock(adapter->q_vector[q_idx]); +#endif napi_enable(&q_vector->napi); } } @@ -1306,6 +1357,12 @@ static void ixgbevf_napi_disable_all(struct ixgbevf_adapter *adapter) for (q_idx = 0; q_idx < q_vectors; q_idx++) { q_vector = adapter->q_vector[q_idx]; napi_disable(&q_vector->napi); +#ifdef CONFIG_NET_RX_BUSY_POLL + while (!ixgbevf_qv_disable(adapter->q_vector[q_idx])) { + pr_info("QV %d locked\n", q_idx); + usleep_range(1000, 20000); + } +#endif /* CONFIG_NET_RX_BUSY_POLL */ } } @@ -1323,31 +1380,55 @@ static void ixgbevf_configure(struct ixgbevf_adapter *adapter) for (i = 0; i < adapter->num_rx_queues; i++) { struct ixgbevf_ring *ring = &adapter->rx_ring[i]; ixgbevf_alloc_rx_buffers(adapter, ring, - IXGBE_DESC_UNUSED(ring)); + ixgbevf_desc_unused(ring)); } } -#define IXGBE_MAX_RX_DESC_POLL 10 -static inline void ixgbevf_rx_desc_queue_enable(struct ixgbevf_adapter *adapter, - int rxr) +#define IXGBEVF_MAX_RX_DESC_POLL 10 +static void ixgbevf_rx_desc_queue_enable(struct ixgbevf_adapter *adapter, + int rxr) { struct ixgbe_hw *hw = &adapter->hw; + int wait_loop = IXGBEVF_MAX_RX_DESC_POLL; + u32 rxdctl; int j = adapter->rx_ring[rxr].reg_idx; - int k; - for (k = 0; k < IXGBE_MAX_RX_DESC_POLL; k++) { - if (IXGBE_READ_REG(hw, IXGBE_VFRXDCTL(j)) & IXGBE_RXDCTL_ENABLE) - break; - else - msleep(1); - } - if (k >= IXGBE_MAX_RX_DESC_POLL) { - hw_dbg(hw, "RXDCTL.ENABLE on Rx queue %d " - "not set within the polling period\n", rxr); - } + do { + usleep_range(1000, 2000); + rxdctl = IXGBE_READ_REG(hw, IXGBE_VFRXDCTL(j)); + } while (--wait_loop && !(rxdctl & IXGBE_RXDCTL_ENABLE)); + + if (!wait_loop) + hw_dbg(hw, "RXDCTL.ENABLE queue %d not set while polling\n", + rxr); + + ixgbevf_release_rx_desc(&adapter->hw, &adapter->rx_ring[rxr], + (adapter->rx_ring[rxr].count - 1)); +} + +static void ixgbevf_disable_rx_queue(struct ixgbevf_adapter *adapter, + struct ixgbevf_ring *ring) +{ + struct ixgbe_hw *hw = &adapter->hw; + int wait_loop = IXGBEVF_MAX_RX_DESC_POLL; + u32 rxdctl; + u8 reg_idx = ring->reg_idx; + + rxdctl = IXGBE_READ_REG(hw, IXGBE_VFRXDCTL(reg_idx)); + rxdctl &= ~IXGBE_RXDCTL_ENABLE; + + /* write value back with RXDCTL.ENABLE bit cleared */ + IXGBE_WRITE_REG(hw, IXGBE_VFRXDCTL(reg_idx), rxdctl); - ixgbevf_release_rx_desc(hw, &adapter->rx_ring[rxr], - adapter->rx_ring[rxr].count - 1); + /* the hardware may take up to 100us to really disable the rx queue */ + do { + udelay(10); + rxdctl = IXGBE_READ_REG(hw, IXGBE_VFRXDCTL(reg_idx)); + } while (--wait_loop && (rxdctl & IXGBE_RXDCTL_ENABLE)); + + if (!wait_loop) + hw_dbg(hw, "RXDCTL.ENABLE queue %d not cleared while polling\n", + reg_idx); } static void ixgbevf_save_reset_stats(struct ixgbevf_adapter *adapter) @@ -1545,8 +1626,6 @@ void ixgbevf_up(struct ixgbevf_adapter *adapter) { struct ixgbe_hw *hw = &adapter->hw; - ixgbevf_negotiate_api(adapter); - ixgbevf_reset_queues(adapter); ixgbevf_configure(adapter); @@ -1679,7 +1758,10 @@ void ixgbevf_down(struct ixgbevf_adapter *adapter) /* signal that we are down to the interrupt handler */ set_bit(__IXGBEVF_DOWN, &adapter->state); - /* disable receives */ + + /* disable all enabled rx queues */ + for (i = 0; i < adapter->num_rx_queues; i++) + ixgbevf_disable_rx_queue(adapter, &adapter->rx_ring[i]); netif_tx_disable(netdev); @@ -1733,10 +1815,12 @@ void ixgbevf_reset(struct ixgbevf_adapter *adapter) struct ixgbe_hw *hw = &adapter->hw; struct net_device *netdev = adapter->netdev; - if (hw->mac.ops.reset_hw(hw)) + if (hw->mac.ops.reset_hw(hw)) { hw_dbg(hw, "PF still resetting\n"); - else + } else { hw->mac.ops.init_hw(hw); + ixgbevf_negotiate_api(adapter); + } if (is_valid_ether_addr(adapter->hw.mac.addr)) { memcpy(netdev->dev_addr, adapter->hw.mac.addr, @@ -1929,6 +2013,9 @@ static int ixgbevf_alloc_q_vectors(struct ixgbevf_adapter *adapter) q_vector->v_idx = q_idx; netif_napi_add(adapter->netdev, &q_vector->napi, ixgbevf_poll, 64); +#ifdef CONFIG_NET_RX_BUSY_POLL + napi_hash_add(&q_vector->napi); +#endif adapter->q_vector[q_idx] = q_vector; } @@ -1938,6 +2025,9 @@ err_out: while (q_idx) { q_idx--; q_vector = adapter->q_vector[q_idx]; +#ifdef CONFIG_NET_RX_BUSY_POLL + napi_hash_del(&q_vector->napi); +#endif netif_napi_del(&q_vector->napi); kfree(q_vector); adapter->q_vector[q_idx] = NULL; @@ -1961,6 +2051,9 @@ static void ixgbevf_free_q_vectors(struct ixgbevf_adapter *adapter) struct ixgbevf_q_vector *q_vector = adapter->q_vector[q_idx]; adapter->q_vector[q_idx] = NULL; +#ifdef CONFIG_NET_RX_BUSY_POLL + napi_hash_del(&q_vector->napi); +#endif netif_napi_del(&q_vector->napi); kfree(q_vector); } @@ -2072,6 +2165,9 @@ static int ixgbevf_sw_init(struct ixgbevf_adapter *adapter) hw->mac.max_tx_queues = 2; hw->mac.max_rx_queues = 2; + /* lock to protect mailbox accesses */ + spin_lock_init(&adapter->mbx_lock); + err = hw->mac.ops.reset_hw(hw); if (err) { dev_info(&pdev->dev, @@ -2082,6 +2178,7 @@ static int ixgbevf_sw_init(struct ixgbevf_adapter *adapter) pr_err("init_shared_code failed: %d\n", err); goto out; } + ixgbevf_negotiate_api(adapter); err = hw->mac.ops.get_mac_addr(hw, hw->mac.addr); if (err) dev_info(&pdev->dev, "Error reading MAC address\n"); @@ -2097,9 +2194,6 @@ static int ixgbevf_sw_init(struct ixgbevf_adapter *adapter) memcpy(hw->mac.addr, netdev->dev_addr, netdev->addr_len); } - /* lock to protect mailbox accesses */ - spin_lock_init(&adapter->mbx_lock); - /* Enable dynamic interrupt throttling rates */ adapter->rx_itr_setting = 1; adapter->tx_itr_setting = 1; @@ -2620,8 +2714,6 @@ static int ixgbevf_open(struct net_device *netdev) } } - ixgbevf_negotiate_api(adapter); - /* setup queue reg_idx and Rx queue count */ err = ixgbevf_setup_queues(adapter); if (err) @@ -3010,7 +3102,7 @@ static int __ixgbevf_maybe_stop_tx(struct ixgbevf_ring *tx_ring, int size) /* We need to check again in a case another CPU has just * made room available. */ - if (likely(IXGBE_DESC_UNUSED(tx_ring) < size)) + if (likely(ixgbevf_desc_unused(tx_ring) < size)) return -EBUSY; /* A reprieve! - use start_queue because it doesn't call schedule */ @@ -3021,7 +3113,7 @@ static int __ixgbevf_maybe_stop_tx(struct ixgbevf_ring *tx_ring, int size) static int ixgbevf_maybe_stop_tx(struct ixgbevf_ring *tx_ring, int size) { - if (likely(IXGBE_DESC_UNUSED(tx_ring) >= size)) + if (likely(ixgbevf_desc_unused(tx_ring) >= size)) return 0; return __ixgbevf_maybe_stop_tx(tx_ring, size); } @@ -3216,6 +3308,8 @@ static int ixgbevf_resume(struct pci_dev *pdev) } pci_set_master(pdev); + ixgbevf_reset(adapter); + rtnl_lock(); err = ixgbevf_init_interrupt_scheme(adapter); rtnl_unlock(); @@ -3224,8 +3318,6 @@ static int ixgbevf_resume(struct pci_dev *pdev) return err; } - ixgbevf_reset(adapter); - if (netif_running(netdev)) { err = ixgbevf_open(netdev); if (err) @@ -3293,6 +3385,9 @@ static const struct net_device_ops ixgbevf_netdev_ops = { .ndo_tx_timeout = ixgbevf_tx_timeout, .ndo_vlan_rx_add_vid = ixgbevf_vlan_rx_add_vid, .ndo_vlan_rx_kill_vid = ixgbevf_vlan_rx_kill_vid, +#ifdef CONFIG_NET_RX_BUSY_POLL + .ndo_busy_poll = ixgbevf_busy_poll_recv, +#endif }; static void ixgbevf_assign_netdev_ops(struct net_device *dev) diff --git a/drivers/net/ethernet/intel/ixgbevf/vf.c b/drivers/net/ethernet/intel/ixgbevf/vf.c index 387b526..4d44d64 100644 --- a/drivers/net/ethernet/intel/ixgbevf/vf.c +++ b/drivers/net/ethernet/intel/ixgbevf/vf.c @@ -242,7 +242,7 @@ static s32 ixgbevf_set_uc_addr_vf(struct ixgbe_hw *hw, u32 index, u8 *addr) msgbuf[0] |= index << IXGBE_VT_MSGINFO_SHIFT; msgbuf[0] |= IXGBE_VF_SET_MACVLAN; if (addr) - memcpy(msg_addr, addr, 6); + memcpy(msg_addr, addr, ETH_ALEN); ret_val = mbx->ops.write_posted(hw, msgbuf, 3); if (!ret_val) @@ -275,7 +275,7 @@ static s32 ixgbevf_set_rar_vf(struct ixgbe_hw *hw, u32 index, u8 *addr, memset(msgbuf, 0, sizeof(msgbuf)); msgbuf[0] = IXGBE_VF_SET_MAC_ADDR; - memcpy(msg_addr, addr, 6); + memcpy(msg_addr, addr, ETH_ALEN); ret_val = mbx->ops.write_posted(hw, msgbuf, 3); if (!ret_val) |