Ref api

[YishaiGlasner]

Summary

• Add a new /api/ref/<tref> endpoint that returns structured metadata for any Sefaria Ref
• Provide consistent navigation and structure metadata across node types (JaggedArrayNode, SchemaNode, Dictionary, Sheets, etc.)
• Avoids redundant MongoDB queries by ensuring a single vstate fetch per request for standard text refs (i.e. not virtual nodes)
• Fix prev_segment_ref / next_segment_ref to correctly handle virtual nodes (e.g. Siddur)
• Add optional state_ja parameter to avoid redundant DB calls when state is already available
• Add a pymongo QueryCounter listener for asserting query counts in tests
• Add comprehensive test coverage and OpenAPI documentation

API Details

New endpoint: GET /api/ref/<tref>

Returns a JSON object with:

• is_ref — whether the input resolves to a valid ref (returns {is_ref: false} for invalid input)
• normalized, hebrew, url_ref — normalized representations
• index_title, node_type — index and node metadata
• depth, address_types, section_names — structure info (for JaggedArrayNode / DictionaryEntryNode)
• start_indexes, start_labels, end_indexes, end_labels — section position
• navigation_refs — contextual navigation:
  • lineage_refs_top_down — ancestor refs from root to parent
  • first_available_section_ref — first section with content
  • first_subref / last_subref — child navigation (non-segment, non-range)
  • prev_section_ref / next_section_ref — section-level navigation
  • prev_segment_ref / next_segment_ref — segment-level navigation
• children — child node titles (for SchemaNode)
• default_child_node — default child metadata when applicable
• sheet_id, lexicon_name, headword — type-specific fields

Considerations

• Navigation scope
  prev_* and next_* are only defined for section-level and segment-level refs.
  Navigation at higher levels is intentionally not exposed to avoid ambiguity. Consumers can traverse upward (via lineage_refs_top_down) and derive such relationships if needed.
• Field presence
  Fields that are not applicable to a given ref type are omitted.
  Fields that are applicable but have no value (e.g. no previous or next ref exists) are returned as null.

Changes in Ref

• Fix prev_segment_ref and next_segment_ref to support DictionaryEntryNode
• Add optional state_ja parameter to selected methods (already supported in others) to improve performance
• Add function get_subrefs_count and refactor all_subrefs, prev_segment_ref and next_segment_ref to use it.

pymongo listener

Adds QueryCounter, a pymongo CommandListener used in tests to:

• Count MongoDB queries per request
• Record query tracebacks for debugging
  Tests reset the counter before each API call and assert on QueryCounter.count. On failure, full query tracebacks are printed to help identify unnecessary database hits.
  The listener is only registered in test environments (sys._called_from_test), so there is zero production overhead.

Note on tests

api/tests.py is currently not part of the CI suite (historical decision).
All new tests were added there and can be run locally.

[Copilot]

Pull request overview

Adds a new GET /api/ref/<tref> API endpoint to validate and introspect Sefaria refs, returning structured node/structure metadata and navigation refs, with accompanying OpenAPI documentation and tests. The PR also updates core Ref navigation helpers to better support virtual nodes and to reduce redundant DB work by allowing callers to pass a pre-fetched VersionState.

Changes:

• Add RefView (/api/ref/<tref>) returning normalized/hebrew/url forms, node metadata, structure fields, and navigation refs.
• Introduce a pymongo QueryCounter listener (test-only) to assert Mongo command counts in API tests.
• Extend/refine Ref navigation/state helpers (prev_segment_ref, next_segment_ref, first_available_section_ref, get_state_ja, is_empty) to accept an optional vstate.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
sefaria/urls_shared.py	Routes the new /api/ref/<tref> endpoint to RefView.
api/views.py	Implements RefView response construction and navigation metadata.
sefaria/model/text.py	Updates Ref navigation + state access to support vstate and virtual-node behavior.
sefaria/system/database.py	Adds QueryCounter and registers it as a pymongo listener in test environments.
api/tests.py	Adds comprehensive tests for the new endpoint + query-count assertions.
docs/openAPI.json	Documents /api/ref/{tref} and the RefJSON response schema.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

is_empty()'s new vstate fast-path is incorrect for many refs. VersionState.state_node(...).ja() returns a JaggedIntArray whose leaf values are ints; calling sub_array_length() after indexing down to a leaf hits the TypeError path and returns 0, which makes segment-level refs (and other fully-specified refs) appear empty even when text exists. Use a content check that works at arbitrary depth (e.g., state_ja.subarray_with_ref(self).is_empty() or get_element() for segment-level) instead of sub_array_length(self.sections).

Suggested change

	return state_ja.sub_array_length([i - 1 for i in self.sections]) in (0, None)
	subarray = state_ja.subarray_with_ref(self)
	return subarray.is_empty()

[Copilot]

In prev_segment_ref() for virtual nodes, when the current ref is the first segment of a section, the previous segment should be the last segment of the previous section. Returning r.all_subrefs()[0] returns the first segment instead (and can also raise IndexError if the previous section has no subrefs). Adjust this to return the last available subref (and handle empty subref lists).

Suggested change

	return r.all_subrefs()[0]
	subrefs = r.all_subrefs()
	if not subrefs:
	# No subrefs available in the previous section; fall back to the section ref itself
	return r
	return subrefs[-1]

[YishaiGlasner]

changing to -1
the assumption that previous section has segments is also when not vurtual.

[YishaiGlasner]

Spec ing ref API and implementing [sc-40554]

[stevekaplan123]

I suggested a possible approach for a re-factor that allows us to avoid checking node type in next_segment_ref or prev_segment_ref. Curious what you think.