Add missing Features for optional and experimental SPKGs

[soehms]

Is there an existing issue for this?

• I have searched the existing issues for a bug report that matches the one I want to file, without success.

Problem Description

According to #35820 (comment) this is needed to eliminate the use of sage.misc.packages from the doctester.

Proposed Solution

Here are the lists we should work through:

A. Optional

1. • biopython
2. • coxeter3, related PRs New modularized distributions sagemath-{sirocco,bliss,meataxe,mcqd,coxeter3,tdlib} #35661, Replace special handling of optional extensions (bliss, coxeter3, ....) #37857
3. • cunningham_tables
4. • cylp
5. • database_kohel
6. • database_mutation_class
7. • database_odlyzko_zeta
8. • database_stein_watkins
9. • database_symbolic_data
10. • fricas, Feature for fricas #33575
11. • frobby
12. • glucose, sage.features.sat #38239
13. • kissat, sage.features.sat #38239
14. • mathics, build/pkgs/mathics: Reduce to a pip package #37395
15. • p_group_cohomology
16. • pari_elldata
17. • pari_galpol
18. • pari_seadata
19. • pycosat, sage.features.sat #38239
20. • pycryptosat, sage.features.sat #38239
21. • pyscipopt
22. • sirocco, related PR New modularized distributions sagemath-{sirocco,bliss,meataxe,mcqd,coxeter3,tdlib} #35661, Replace special handling of optional extensions (bliss, coxeter3, ....) #37857
23. • snappy
24. • topcom, sage.features.topcom #37858

B. Experimental

1. • gambit, gambit: Re-create as a pip package pygambit (help wanted!) #37809
2. • gap3
3. • lie
4. • qepcad
5. • symengine_py

C. Standard

These standard packages still have optional tags which should be removed changed to # needs tags (– mkoeppe):

1. • beautifulsoup4
2. • rpy2, Activate/fix rpy2 tests #35396 changes # optional to # needs

Alternatives Considered

The purpose of this issue is to track the progress on this task (similar to a Trac Meta-Ticket).

Related PR and issues

• Meta-ticket: Create upstream repositories, pip-installable packages for database packages #30914
• Replace have_program usage by Features #32957
• Doctester: Issue a warning when unknown # optional / # needs tags are used #38217
• sage.databases: Use sage.feature to define database file system location #30819

Additional Information

The above list of packages has been detected by the following script:

from sage.misc.package import  list_packages
from sage.features.all import all_features
from sage.env import SAGE_SRC
from os import environ, chdir, path
from subprocess import check_output, CalledProcessError
import re


pwd = environ['PWD']

spkgs = list_packages(local=True)

sl = set([k for k, v in spkgs.items() if v.type == 'standard'])
ol = set([k for k, v in spkgs.items() if v.type == 'optional'])
el = set([k for k, v in spkgs.items() if v.type == 'experimental'])

af_list = list(all_features())
af_spkg = set([f.spkg for f in af_list if f.spkg is not None])
af_name = set([f.name for f in af_list])
af = af_spkg.union(af_name)
af.add('gap_packages') # manually added since no individual instance of GapPackage is in the af_list

sl_minus_af = sl.difference(af)
ol_minus_af = ol.difference(af)
el_minus_af = el.difference(af)

sage_src = path.join(SAGE_SRC, 'sage')
chdir(sage_src)

def test_doctest_tag(spkg_name):
    cmd = 'git grep \\#[\\ ]*optional.*%s | grep -v "doctest/" | grep -v "tests/cmdline"' % spkg_name
    try:
        res = check_output(cmd, shell=True).decode()
        if re.match(r'^[^\n]*#[\s]+optional[^\n]*[\s]%s[\s,\n]' % spkg_name, res) is None:
            return False
        return True
    except CalledProcessError:
        pass
    return False

missing_standard_features = sorted([f for f in sl_minus_af if test_doctest_tag(f)])
missing_optional_features = sorted([f for f in ol_minus_af if test_doctest_tag(f)])
missing_experimental_features = sorted([f for f in el_minus_af if test_doctest_tag(f)])

chdir(pwd)

print('missing_standard_features', missing_standard_features)
print('missing_optional_features', missing_optional_features)
print('missing_experimental_features', missing_experimental_features)

[soehms]

For a start I would take the databases, mathics, snappy and gap3.

[mkoeppe]

Thanks for opening this issue! For reference, here are a few related issues:

• Feature for fricas #33575
• Meta-ticket: Create upstream repositories, pip-installable packages for database packages #30914
• Replace have_program usage by Features #32957

[soehms]

For reference, here are a few related issues ...

Thanks! I added these to the header.