Initiative B Triage: Inlined-Former-Vendor Modules¶
This document captures the Initiative B triage decisions for inlined-former-vendor modules under pipenv/utils/. Each section flagged a symbol or symbol-group as adopt, vendor, or delete. The work is executed directly under Initiative B / Wave 1c rather than tracked as GitHub issues — the per-symbol audit below is the working record.
Scope covered by this synthesis:
pipenv/utils/requirementslib.py(T_B.1) — 20 public symbolspipenv/utils/requirements.py(T_B.2) — 9 functions +BAD_PACKAGESconstantpipenv/utils/fileutils.py(T_B.3) — 3 URL/path converterspipenv/utils/markers.py(T_B.4) — no separate vendor lineage; recorded for completeness
Summary table¶
Rows are sorted by Decision (delete, then vendor, then adopt — cheapest first), and alphabetically within each decision group. Caller counts are external call sites; see the per-module sections for grep methodology.
Symbol or group |
Module |
Decision |
Caller count |
|---|---|---|---|
|
|
delete |
0 (cascade from |
|
|
delete |
0 (shadowed by canonical copy in |
|
|
delete |
0 |
|
|
delete |
0 |
|
|
delete |
0 (shadowed by canonical copy in |
|
|
delete |
4 (callers migrate to |
|
|
investigate (was: delete) |
1 ( |
|
|
delete |
0 |
|
|
replace (purpose-built helper) |
0 external (used by |
|
|
replace (purpose-built helper) |
0 external (used by |
|
|
replace (purpose-built helper) |
0 external (used by |
|
|
replace (purpose-built helper) |
0 external (one internal use in |
|
|
replace (purpose-built helper) |
0 external (used by |
|
|
replace (purpose-built helper) |
0 external (used by |
|
|
adopt (with provenance docstring) |
0 external (only by |
|
|
adopt (with provenance docstring) |
0 external (only by |
|
|
adopt |
2 |
|
|
adopt |
2 |
|
|
adopt (or delete via patched-pip) |
1 (only internal: |
|
|
adopt |
2 |
|
|
adopt |
2 |
|
|
adopt (collapse duplicate in |
2 import sites + 4 use sites |
|
|
adopt |
0 external (gates |
|
|
adopt |
4 import sites + multiple use sites |
|
|
adopt (rebuilt on purpose-built helper) |
3 (project’s public entry point) |
|
|
adopt |
0 external (only internal caller is |
|
|
adopt |
3 |
|
|
adopt |
0 external (only by |
|
|
adopt |
2 |
|
|
adopt |
1 |
|
|
adopt (or delete via patched-pip) |
2 (verbatim copy of |
|
|
adopt |
2 (in |
The markers.py module is not in this table: T_B.4 found it is owned project glue over pipenv.patched.pip._vendor.{distlib,packaging} and has no separate vendor lineage to triage. It is recorded under its own section below so the “is this vendored?” question stays closed.
Cross-cutting flags surfaced during triage (acted on under other initiatives, not this one):
normalize_name↔pep423_nameoverlap — Initiative E (requirement-model consolidation).Second
path_to_urlatpipenv/utils/shell.py:104— Initiative A’s URL/scheme consolidation pass.BAD_PACKAGESconstant inrequirements.py— Initiative E co-location question.
pipenv/utils/requirementslib.py (T_B.1)¶
Per-symbol audit of the 740-line module pipenv/utils/requirementslib.py,
inlined from the now-archived requirementslib PyPI package. For each
public top-level symbol the table records: a one-line purpose; the count
of internal callers (excluding the definition site, excluding
patched//vendor/); a provenance assessment; and a recommendation of
adopt (project owns it; refactor freely), vendor (treat as frozen
upstream; move under pipenv/vendor/ and stop modifying), or delete
(no internal callers, no plan to expose externally).
Method. Caller counts come from
grep -rn "\b<symbol>\b" pipenv/ --include="*.py" | grep -v patched | grep -v vendor, with the definition site removed and ambiguous matches (e.g. attribute accesses likelink.is_vcs,sysconfig.get_path) audited by hand. The orchestrator can re-verify any row by running the same grep.
Per-symbol table¶
# |
Symbol |
Purpose (one line) |
Internal callers |
Provenance |
Recommendation |
|---|---|---|---|---|---|
1 |
|
Rewrite |
0 (no importers, no in-module callers) — delete candidate |
Pip-internal-style VCS URI helper; mirrors logic in legacy |
delete |
2 |
|
Inverse of #1: add |
2 ( |
Pip-internal-style VCS URI helper. Not boltons. |
adopt |
3 |
|
Return True if a Pipfile entry (mapping or string) is a VCS dep. |
4 import sites ( |
Project predicate over Pipfile schema — pipenv-specific. Not from a live upstream. |
adopt |
4 |
|
Return True if a Pipfile entry has |
2 import sites ( |
Project predicate. Not boltons. |
adopt (also: collapse the dependencies.py duplicate during the adopt pass) |
5 |
|
Return True if a Pipfile version is |
0 importers of this |
Project predicate. The dependencies.py copy is the canonical one. |
delete |
6 |
|
Convert a Pipfile |
1 (only internal: |
Pipenv-specific helper over Pipfile schema. |
delete (only kept alive by #7, which is itself a delete candidate — cascades) |
7 |
|
Heuristic: is the given path/URL something pip can install? |
0 importers — delete candidate |
Pip-internal-style predicate; legacy of |
delete |
8 |
|
Locate |
0 importers — delete candidate |
Legacy of |
delete |
9 |
|
Build |
1: |
Pipenv-specific. The two copies diverge: the |
investigate under W3 — pair with the |
10 |
|
Thin shim around |
0 importers — delete candidate. A near-duplicate at |
Pip-internal-style shim. The resolver.py copy is canonical. |
delete |
11 |
|
Exception type unifying |
0 external; used internally by |
boltons.iterutils (live upstream). |
vendor (group A — see below) |
12 |
|
Look up a value in a nested dict/list by tuple path, with optional default. |
0 external. The two |
boltons.iterutils (live upstream). |
vendor (group A) |
13 |
|
No-op visit callback for |
0 external; used by |
boltons.iterutils (live upstream). |
vendor (group A) |
14 |
|
|
0 external; used by |
boltons.iterutils (live upstream); modified to add |
vendor (group A) — but flag the tomlkit-specific branches as a pipenv-side modification a maintainer may need to keep applied if/when this is re-vendored from upstream boltons. |
15 |
|
|
0 external; used by |
boltons.iterutils (live upstream); modified for |
vendor (group A) — same tomlkit-modification caveat as #14. |
16 |
|
Recursive traverse-and-transform over heterogeneous nested structures. |
0 external; used by |
boltons.iterutils (live upstream). |
vendor (group A) |
17 |
|
Merge a list of nested dicts (Pipfile categories) using |
3 import sites ( |
Built on top of the boltons primitives but the signature (Pipfile-category list merge) is pipenv’s own usage shape. |
adopt (this is the project’s public entry-point into the boltons tree) |
18 |
|
Construct pip’s |
2 ( |
Pipenv-specific thin wrapper around |
adopt |
19 |
|
Unpack a |
2 ( |
Lifted verbatim from |
adopt — but flag for follow-up: investigate whether the patched-pip copy can be called directly. |
20 |
|
Download an HTTP URL into a temp dir; honour |
1 (only internal: |
Lifted verbatim from |
adopt (lives with #19; if #19 is later swapped for the patched-pip version, this dies with it) |
Coherent groups¶
Group A — Boltons iterutils dict-tree walkers (vendor as a unit)¶
PathAccessError, get_path, default_visit, dict_path_enter,
dict_path_exit, remap (rows 11–16). These six symbols are an inlined,
lightly-modified copy of boltons.iterutils. The package is still
maintained upstream (https://github.com/mahmoud/boltons). They have zero
non-pipenv external callers and only one in-tree caller: merge_items
(row 17), which is pipenv-owned.
Recommendation: vendor the boltons subset under pipenv/vendor/boltons/
via the project’s standard vendoring tooling, then have merge_items
import from there. This stops the maintenance treadmill on code we
should not be modifying. The two pipenv-side modifications worth
preserving on re-vendor are the tomlkit.items.Table/InlineTable/
Array branches inside dict_path_enter (row 14) and dict_path_exit
(row 15) — these exist so merge_items works on tomlkit’s container
types directly, not just plain dicts/lists.
Group B — Resurrected pip prepare-step helpers (adopt as a pair)¶
unpack_url and get_http_url (rows 19–20). Lifted verbatim from
pip._internal.operations.prepare. They form a closed pair (get_http_url
has no external callers; it exists solely to support unpack_url).
Recommendation: adopt the pair for now, but log a follow-up to
investigate whether pipenv.patched.pip._internal.operations.prepare.unpack_url
can be called directly. If yes, both rows die together.
Group C — Pipfile-schema project predicates (adopt)¶
is_vcs, is_editable, add_ssh_scheme_to_git_uri, merge_items,
get_pip_command (rows 2, 3, 4, 17, 18). These are pipenv-specific. The
domain is “the shape of a Pipfile entry” plus “where pip’s options
parser lives”, which is pipenv’s responsibility regardless of upstream
status. They should be refactored freely. During the adopt pass:
Collapse the
is_editableduplicate atdependencies.py:1503. Inspect whether the missing-estring branch is intentional in any callers.Decide where these helpers actually belong long-term —
requirementslib.pyis a historical name;pipenv/utils/pipfile_schema.pyor absorbing them intopipenv/utils/pipfile.pywould be more honest.
Group D — Dead code (delete candidates)¶
Rows 1, 5, 6, 7, 8, 9, 10. All have zero importers in pipenv/
(after disambiguating attribute-access collisions and accounting for
in-module-only callers that themselves have no importers):
strip_ssh_from_git_uri(row 1) — never called.is_star(row 5) — shadowed by the canonical copy atdependencies.py:418.convert_entry_to_path(row 6) — only kept alive byis_installable_file, which is itself dead.is_installable_file(row 7) — never imported.get_setup_paths(row 8) — never imported.prepare_pip_source_args(row 9) — shadowed by the canonical (and more robust) copy atindexes.py:18.get_package_finder(row 10) — shadowed by the canonical copy atresolver.py:162.
Recommendation: these are the cheap wins for Initiative E (dead-code
removal). Deleting all seven (plus convert_entry_to_path as a cascade
from is_installable_file) drops roughly 180 lines from the module
without behaviour change. The prepare_pip_source_args and
get_package_finder rows in particular are doubly compelling — they are
divergent stale copies, so removing them eliminates a future
“which-version-am-I-importing?” footgun.
Summary (requirementslib.py)¶
Bucket |
Count |
Symbols |
|---|---|---|
adopt |
7 |
|
vendor |
6 |
|
delete |
7 |
|
total |
20 |
Most consequential findings (requirementslib.py)¶
Seven of twenty public symbols are dead code or stale duplicates of canonical copies elsewhere in the tree. This is the largest cheap win identified so far in the modernization sweep — likely ~25% of the module by line count.
Two of those seven dead duplicates (
prepare_pip_source_args,get_package_finder) have divergent implementations from the versions actually in use. Anyone editing this module today is plausibly editing the wrong copy without noticing. Delete-first is the right hygiene step before any further work on Initiative B.The six boltons-derived dict-walkers cluster cleanly into a single
vendordecision. They have a single in-tree user (merge_items), which makes the redirect mechanical. The only wrinkle is that pipenv has added tomlkit-container branches insidedict_path_enter/dict_path_exit; if these are kept after vendoring, the modifications need to live as a small adapter, not as edits to the vendored file.is_editableis duplicated within the project (requirementslib.pyvs.dependencies.py:1503) with different semantics — the dependencies.py copy drops the-estring-prefix branch. The refactor pass should reconcile these intentionally rather than letting two slightly-wrong copies persist.unpack_url/get_http_urlare verbatim copies of still-extantpip._internal.operations.preparehelpers. The refactor pass should evaluate whether the patched-pip versions can be called directly; if so the adopt-count drops from 7 to 5.
pipenv/utils/requirements.py (T_B.2)¶
Per-function audit of the 395-line module. Caller counts are external
call sites only (the symbol’s own definition and intra-module helper
calls are excluded). Greps were run across pipenv/, excluding
pipenv/patched/ and pipenv/vendor/.
Reproduce a count with, e.g.:
grep -rn "normalize_name" pipenv/ --include="*.py" \
| grep -v pipenv/patched/ | grep -v pipenv/vendor/
Function table¶
# |
Symbol |
One-line description |
External callers |
Provenance |
Recommendation |
|---|---|---|---|---|---|
1 |
|
Mask credentials in a URL netloc, preserving |
0 (only called by |
Pip-internal fork. Pip’s |
vendor (keep as project-owned fork; see provenance note below) |
2 |
|
Apply |
1 ( |
Pip-internal fork. Pip’s |
vendor (keep as project-owned fork; tied to #1) |
3 |
|
Lowercase a name and replace |
4 call sites: |
Project-owned, overlaps with |
delete (move/merge under Initiative E — see overlap section) |
4 |
|
Parse a |
2 ( |
Project-owned bridge: composes pip parsing with the |
adopt |
5 |
|
Decide whether a Pipfile index requires HTTPS based on the trusted-hosts list, then delegate to |
2 ( |
Project-owned bridge. Name collides with the method |
adopt (consider renaming under Initiative E to disambiguate from the |
6 |
|
Convert a single |
3 ( |
Project-owned lockfile bridge. |
adopt |
7 |
|
Map |
2 ( |
Project-owned lockfile bridge. |
adopt |
8 |
|
Convert a single Pipfile entry (str or dict) into a pip-installable line using Pipfile version specifiers rather than locked versions. |
0 (only invoked by the sibling |
Project-owned Pipfile bridge. |
adopt |
9 |
|
Map |
1 ( |
Project-owned Pipfile bridge. |
adopt |
The module-level constant BAD_PACKAGES (line 148) is not a function
but is also a public export and is imported by pipenv/routines/graph.py,
pipenv/routines/clean.py, and pipenv/routines/uninstall.py. It is in
scope for Initiative E only as a co-location question and is not part of
this audit’s adopt/vendor/delete tally.
The normalize_name / pep423_name overlap (flag for Initiative E)¶
The two helpers normalize a distribution name to the same target (lowercase, hyphenated) and live in sibling modules:
pipenv/utils/requirements.py:61
def normalize_name(pkg) -> str:
"""Given a package name, return its normalized, non-canonicalized form."""
return pkg.replace("_", "-").lower()
pipenv/utils/dependencies.py:134
def pep423_name(name):
"""Normalize package name to PEP 423 style standard."""
name = name.lower()
if any(i not in name for i in (VCS_LIST + SCHEME_LIST)):
return name.replace("_", "-")
else:
return name
For ordinary distribution names the two functions return the same
string. pep423_name adds a guard that skips the _→- rewrite when
the input contains every VCS scheme and URL scheme in
VCS_LIST + SCHEME_LIST simultaneously — a condition that is effectively
unreachable for real package names, which means the two helpers are de
facto equivalent in practice. The guard is also almost certainly buggy:
the any(... not in name for ...) predicate is true for any name that
is missing at least one scheme token, so the early-return branch is the
common case and the else branch is dead. That bug is out of scope for
this triage.
External call-site counts:
normalize_name: 4 external call sites (table above).pep423_name: 11 external call sites acrosspipenv/utils/locking.py,pipenv/project.py,pipenv/routines/outdated.py, andpipenv/routines/uninstall.py.
Recommendation (flag, do not act here): normalize_name is a
candidate to be merged into pep423_name (or both replaced by a single
canonical helper, plausibly pipenv.utils.dependencies.normalize_name)
under Initiative E — requirement-model consolidation. This triage
only flags the overlap; the actual move belongs to Initiative E so it
can be planned alongside the other dependency-helper relocations.
Why we don’t just from pip._internal.utils.misc import redact_*¶
pipenv/utils/requirements.py:18 and :52 look at first glance like
copy-paste of pipenv/patched/pip/_internal/utils/misc.py:456 and
:523. They are not safe to delete in favor of the pip imports:
Env-var placeholders are preserved. Pip’s
redact_netlocunconditionally rewrites the user (and password) to****. Pipenv’s variant matches${\w+}on both fields and leaves the placeholder intact. Pipenv users routinely encode credentials with env-var expansion in their Pipfile/lockfile (e.g.https://${PYPI_USER}:${PYPI_PASS}@example.com/simple); redacting those placeholders would lose information the user expects to see after a round-trip.Standard SSH usernames are preserved. A
STANDARD_SSH_USERNAMES = ("git",)tuple keepsgit@github.com:org/repo.gitreadable instead of rewriting it to****@github.com:org/repo.git. Pip has no equivalent.
Both behaviors are user-visible (they appear in CLI output and in the
generated Pipfile.lock), so swapping in pip’s version would be a
behavior change rather than a refactor. The recommendation is therefore
vendor (treat as a project-owned fork) rather than adopt-by-import.
A # Fork of pip._internal.utils.misc.{redact_netloc,redact_auth_from_url}
docstring note would make the divergence obvious to future readers; that
small comment cleanup belongs to the cleanup pass that actually edits
these functions, not to this triage.
Summary (requirements.py)¶
Recommendation |
Count |
Symbols |
|---|---|---|
adopt |
6 |
|
vendor |
2 |
|
delete |
1 |
|
total |
9 |
Key findings to carry forward:
The
normalize_name↔pep423_nameoverlap is the one cross-module duplicate in this file. It belongs to Initiative E’s requirement-model consolidation; flagged here so E can plan the merge.The two redact helpers are deliberate forks, not blind copies; the cleanup pass should add a one-line provenance comment and leave the behavior alone.
The four pipfile/lockfile bridge functions (
*_from_lockfile,*_from_pipfile) are project-owned glue between pip’s parser and pipenv’s lockfile/Pipfile schemas; they have no upstream equivalent and should be kept as-is for this initiative.BAD_PACKAGESis also exported from this module and used by threepipenv/routines/files; flagging for Initiative E in case the constant moves with the helpers.
URL/path helpers in pipenv/utils/fileutils.py (T_B.3)¶
Narrow audit of the three URL/path converters in
pipenv/utils/fileutils.py: is_file_url, url_to_path, path_to_url.
is_valid_url also lives in this file but is the duplicate flagged in
Initiative A (T_A.2) and is out of scope here.
Domain-boundary rule¶
Initiative A draws the line as: URL/scheme concerns belong in
pipenv/utils/internet.py; filesystem-path concerns belong in
pipenv/utils/fileutils.py. The three symbols below all sit on the
boundary because they translate between the two. Under the rule, the
deciding question is “which side is doing the heavy lifting?”. For these
three, the answer is the filesystem side — they exist specifically to
move a Path across the file:// boundary, and only the file: scheme
is meaningful to them. A generic URL utility that knew nothing about
local paths could not implement them. So they stay in fileutils.py.
Per-symbol recommendation¶
is_file_url— Keep infileutils.py. It is a scheme check, but it exists only to gate thefile://-vs-everything-else branch inopen_file,url_to_path, and (in callers) path-vs-URL dispatch. No external (non-test) callers inpipenv/outside this module today.url_to_path— Keep infileutils.py. Returns apathlib.Path, handles UNC netloc reconstruction, and is the inverse ofpath_to_url. External callers:pipenv/utils/requirementslib.py(2 sites).path_to_url— Keep infileutils.py. Operates on aPath, callsnormalize_drive(also infileutils.py), and emits afile://URI. Only internal caller today isopen_filein the same module.
Adjacent finding (not in scope, flagged for later)¶
pipenv/utils/shell.py:104 defines a second path_to_url with a
different implementation (Path(...).as_uri() vs. the quoting-aware
version here) and no callers in pipenv/. This is a duplicate-name
hazard analogous to the is_valid_url case in Initiative A; recommend
folding into Initiative A’s URL/scheme consolidation pass rather than
opening a new task here.
pipenv/utils/markers.py (T_B.4)¶
pipenv/utils/markers.py is owned project glue, not inlined former vendor
code. Its import head (lines 1-14) pulls parse_marker from
pipenv.patched.pip._vendor.distlib.util and InvalidMarker, Marker,
Specifier, and SpecifierSet from pipenv.patched.pip._vendor.packaging;
the marker/specifier semantics themselves live in those vendored libraries
and are managed by the vendor tooling, while this module only composes them
into pipenv-specific helpers (cleanup, intersection, lookup tables such as
MAX_VERSIONS / DEPRECATED_VERSIONS, and the local RequirementError).
There is therefore no separate vendor lineage to triage under Initiative B,
and the module can be refactored freely under Initiative E (requirement-model
consolidation) if useful. Recorded here so the “is this vendored?” question
stays closed.
