Modify CI tests to use bf-pktpy module, with no scapy, except for EBPF backend tests

[jafingerhut]

All prerequisite PRs have been approved and merged, so this PR is ready for review now. All changes to requirements.txt to point at their final "official" commit SHAs are in this PR now.

With these changes, 1 out of the 7 remaining test Python programs that uses Scapy now no longer uses Scapy, leaving only 6 that I believe would take significantly more work to update, and I don't have any plans to change them any time soon. A separate PR proposes licensing those remaining 6 EBPF back end Python test programs as license GPL-2.0-only.

Also with these changes, only the EBPF CI test uses the Scapy library. The rest of the CI tests DO NOT install scapy at all, and they pass. This is good for ensuring that no one accidentally slips in new code that depends on Scapy in the future.

The following prerequisites have already been merged:

• Add bf pktpy source ptf#221
  • This older PR is unnecessary if the one above is merged in: Enable more common bf-pktpy and scapy names to be used through ptf package open-p4studio#85
• Add docs on using pmm (packet manipulation module) bf_pktpy, and option to select pmm by env variable ptf#213
• Replace unnecessary use of Scapy with ptf.pcap_writer in p4tc tests #5165

[jafingerhut]

Since originally writing this comment, I have made the ptf-ebpf tests by installing scapy, but only for the ptf-ebpf tests. I have verified that scapy is not installed for any of the other tests, only bf-pktpy.

@fruffy I am not sure whether there would be any interest in checking in changes similar to this, but basically the all-tests-pass-except the ptf-ebpf test demonstrates that if you do not have scapy installed on your system at all, but you do have the bf-pktpy module installed (which is licensed Apache-2.0), it implements enough functionality that all tests work, both those that do not use ptf at all, and those that do (except the ptf-ebpf tests).

The reason that ptf-ebpf tests fail here is that they import the ptf module, and they import and use scapy directly, too. Some of those ptf-ebpf tests using scapy appear to me unlikely to work with bf-pktpy as an attempted replacement, without probably-significant enhancements to bf-pktpy. I have not attempted to make such enhancements to bf-pktpy, nor do I plan to.

The other passing tests that do use ptf typically only use a handful of functions in ptf, and I believe that only the testgen-p4c-bmv2-ptf tests import the module ptf.packet, which is the only sub-module of ptf that causes either scapy or bf-pktpy to be used. For those tests, bf-pktpy is a compatible replacement, primarily because they do not use much (if any) scapy functionality, and it is a small subset that bf-pktpy does implement. I am not certain, but it appears that the only parts of module ptf that p4testgen PTF tests use are these:

ptf.ptfutils.test_param_get
ptf.ptfutils.send_packet
ptf.ptfutils.verify_packet
ptf.ptfutils.verify_no_other_packets

All packets sent in, or verified as output packets, by testgen-generated PTF tests give the contents of the packets as raw byte strings, so there is no use of any Scapy classes like Ether, IP, IPv6, TCP, UDP, etc. p4testgen is written to be generic to whatever the parser definition is in the P4 program it is generating tests for, so it would be more work to try to use those features of scapy than not to.

[jafingerhut]

One possible reason to consider committing changes like these, is that they would be likely to catch any changes someone tried to make that added dependencies on scapy -- such tests would fail, unless they were part of the ebpf tests.