SWE-QA: A Dataset and Benchmark for Complex Code Understanding Laïla Elkoussy, Julien Perez∗,† LRE, EPITA* [email protected]
Bpifrance [email protected] †
arXiv:2604.24814v1 [cs.SE] 27 Apr 2026
Abstract In this paper, we introduce SWE-QA, a text and code corpus aimed at benchmarking multi-hop code comprehension, addressing the gap between simplified evaluation tasks and the complex reasoning required in real-world software development. While existing code understanding benchmarks focus on isolated snippets, developers must routinely connect information across multiple dispersed code segments. The dataset comprises 9,072 multiple-choice questions systematically generated from 12 Python repositories of SWE-bench, evaluating several recurrent reasoning patterns like Declaration-and-Call questions that link entity definitions to their usage, and Interacting-Entity questions that examine the dynamic relationships among multiple collaborating components. Generated through parsing-based entity extraction and Large Language Model assisted question construction with carefully validated distractors, the benchmark distinguishes genuine comprehension from superficial pattern matching. Evaluation of 15 language models (360M to 671B parameters) reveals significant challenges in multi-hop reasoning, with best performance reaching 74.41% accuracy. Dense architectures consistently outperform mixture-of-experts models by 10-14 percentage points, while reasoning-enhanced variants show inconsistent benefits. Keywords: Corpus (Creation, Annotation, etc.), Information Extraction, Information Retrieval, Question Answering
1.
Introduction
Code comprehension remains a fundamental challenge in natural language processing for software engineering, with implications extending from academic research to practical software development and automated programming assistance. While significant progress has been made in developing language models for code understanding, capturing syntax, static semantics, and even dynamic behavior (Ma et al., 2023), existing evaluation frameworks predominantly focus on localized reasoning tasks. Such frameworks often fail to capture the complex, project-scale, interconnected nature of real-world software systems, where multi-hop reasoning, interprocedural dependencies, and runtime behavior matter deeply, which is essential for code development agents (She et al., 2023; Zhang et al., 2023). Current benchmarks typically evaluate models on isolated code snippets or self-contained problems where all necessary information is available within a single context. This misrepresents how software engineers interact with large codebases. Real-world software understanding requires reasoning across multiple, disparate code segments, a process similar to multi-hop reasoning in reading comprehension tasks like HotpotQA (Yang et al., 2018), but with added complexity from syntax, execution semantics, and dependencies. Consider a typical developer workflow: encoun-
tering a function call in one file, navigating to its definition in another file to understand parameters and behavior, then tracing how returned values are used elsewhere across modules. This requires synthesizing information from multiple sources, understanding temporal dependencies, and reasoning about data flow patterns, skills that current benchmarks largely fail to assess. Multi-hop code comprehension arises in linking function declarations to invocations across files, tracing class hierarchies, following data pipelines, understanding event-driven architectures, and comprehending complex object interactions. Despite its importance, existing datasets provide insufficient coverage. Benchmarks like CodeQA, CS1QA, and CodeXGLUE primarily evaluate singlehop reasoning within contained contexts. Even sophisticated benchmarks like SWE-bench, while operating on real repositories, focus on single-issue resolution rather than explicit multi-entity tracking. This gap limits assessment of language models for real-world software engineering. Multi-hop reasoning over large codebases also highlights a limitation of current large language models: finite context windows. Even with extended context, models often struggle to maintain coherence when given excessive or poorly scoped information (Liu et al., 2023; Xiao et al., 2024). This underscores the need for evaluation settings that control the amount and structure of context, ensuring performance reflects
lib/mpl toolkits/axes grid1/axes size.py - Entity Definition 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
class Scaled(_Base): """ Simple scaled(?) size with absolute part = 0 and relative part = *scalable_size*. """ def __init__(self, scalable_size): self._scalable_size = scalable_size def get_size(self, renderer): rel_size = self._scalable_size abs_size = 0. return rel_size, abs_size Scalable = Scaled def _get_axes_aspect(ax): aspect = ax.get_aspect() if aspect == "auto": aspect = 1. return aspect
galleries/examples/axes grid1/demo fixed size axes.py - Entity Usage 1 2 3 4 5 6 7 8 9 10
# The first & third items are for padding and the second items are for the # Axes. Sizes are in inches. h = [Size.Fixed(1.0), Size.Scaled(1.), Size.Fixed(.2)] v = [Size.Fixed(0.7), Size.Scaled(1.), Size.Fixed(.5)] divider = Divider(fig, (0, 0, 1, 1), h, v, aspect=False) # The width and height of the rectangle are ignored. ax = fig.add_axes(divider.get_position(), axes_locator=divider.new_locator(nx=1, ny=1)) ax.plot([1, 2, 3]) plt.show()
Question Metadata Repo: matplotlib/matplotlib Question ID: 133 Category: entity declaration call specific Entity: Scaled Correct Answer: C MCQ Question Question: How does the Scaled class handle the case where the absolute part of the size is zero, and what implications does this have for the plot’s layout? A) When the absolute part of the size is zero, the Scaled class returns abs size = 1, ensuring that the plot always has a minimum size. B) The Scaled class handles the case where the absolute part of the size is zero by setting it to None, effectively removing the size from the plot. C) When the absolute part of the size is zero, the Scaled class returns abs size = 0, effectively ignoring the absolute part of the size. This can lead to distorted plots, especially when the aspect ratio is not suitable for the data being represented. D) The Scaled class uses a heuristic to handle the case where the absolute part of the size is zero, scaling the size to a default value of 1.0.
Figure 1: Multi-hop question sampled from SWE-QA requiring cross-file reasoning: the Scaled class in axes_size.py (top left) and its use in demo_fixed_size_axes.py (bottom left). A correct answer must trace how abs_size = 0 in get_size affects layout when Size.Scaled(1.) is invoked. The multiple-choice design uses targeted distractors to distinguish true cross-context understanding from superficial pattern matching, mirroring the multi-step reasoning developers perform in real codebases. reasoning ability rather than context length. We address this gap by introducing a novel dataset and benchmark1 designed to evaluate multihop, repository-scale code comprehension. Our approach systematically constructs questions requiring understanding of complex relationships between code entities across different file segments, mimicking the reasoning processes developers use in large-scale software systems. We focus on two fundamental categories: declaration-and-call relationships connecting entity definitions with usage contexts, and interacting entity relationships requiring understanding of how multiple components collaborate. By examining multi-hop scenarios from real repositories, we bridge the gap between simplistic evaluation tasks and real software comprehension, offering an authentic assessment of the cognitive challenges faced by developers and automated systems. Finally, we leverage a systematic methodology for generating authentic multi-hop code comprehension questions from real software repositories, ensuring relevance, quality, and diversity. Based on this, we curate a dataset with two categories of multi-hop questions and carefully designed distractors, providing a challenging benchmark that tests true understanding rather than memorization. We evaluate multiple state-of-the-art language models on this benchmark, revealing significant performance gaps and highlighting strengths and weaknesses in handling complex code relationships, with attention to scaling effects on multi-hop reasoning. We analyze common failure patterns and 1 Dataset available at: https://github.com/ lailanelkoussy/swe-qa
error types, offering actionable insights for improving model design and training strategies. Finally, we explore the relationship between model size, architecture, and reasoning performance, providing guidance for practitioners optimizing language models for complex software engineering tasks. Our contributions are threefold. First, we present SWE-QA, a curated dataset with diverse, highquality multi-hop questions and carefully designed distractors, enabling robust assessment of genuine code understanding beyond superficial pattern matching. Second, we introduce a systematic methodology for generating this dataset from real repositories, ensuring practical relevance. Third, we evaluate state-of-the-art language models on this benchmark, revealing performance gaps, common failure patterns, and insights for improving multi-hop reasoning over code.
2.
Related Work
Code comprehension research has primarily focused on single-hop reasoning within isolated code snippets or simple queries. CodeQA (Liu et al., 2021) provides Java and Python Q&A pairs from snippets, while CS1QA (Sohn et al., 2022) collects Q&A from introductory programming courses. These datasets emphasize local understanding without requiring reasoning across dispersed code elements. CRUXEval (Gu et al., 2024) contains 800800 800 short Python functions for assessing reasoning, understanding, and execution through two tasks: CRUXEval-I (input prediction) and CRUXEval-O (output prediction). Each function, generated with Code Llama 34B and filtered for
human solvability within a minute, includes inputoutput examples. Although it tests execution tracing more deeply than prior benchmarks, it remains limited to isolated function-level reasoning. Code World Models (team et al., 2025) train language models on observation-action trajectories from Python environments to improve execution understanding through world modeling, yet they do not address tracing dependencies across multiple code segments in large repositories. Broader benchmarks such as CodeXGLUE (Lu et al., 2021), HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and SWE-bench (Jimenez et al., 2024) evaluate realistic programming tasks like completion, translation, and patch generation, but still lack multi-element reasoning. LocAgent (Chen et al., 2025) addresses the related problem of code localization—identifying where in a codebase changes must be made—by parsing repositories into directed heterogeneous graphs that capture files, classes, functions, and their dependencies (imports, invocations, inheritance), and leveraging LLM agents to navigate these graphs via multi-hop reasoning. While LocAgent demonstrates strong performance on file-level localization and downstream issue resolution, it is task-specific and does not provide a comprehension benchmark for evaluating cross-context understanding across diverse reasoning patterns. In natural language processing, multi-hop datasets such as HotpotQA (Yang et al., 2018), MuSiQue (Trivedi et al., 2022), WikiHop, and WikiMultihopQA (Welbl et al., 2018) combine information across sources. Applying this to code is difficult due to syntax, dependencies, and execution semantics. Table (?) in Appendix A summarizes the key dimensions along which existing benchmarks differ, highlighting the gap that SWE-QA is designed to fill. Our work bridges this gap by building a multi-hop code comprehension dataset that extends multi-step reasoning to software repositories, evaluating models on tracing logical dependencies and entity interactions across repository-scale codebases.
3.
Methodology
Our approach comprises three steps: code processing and chunking, multi-hop question generation, and quality control through post-processing and benchmarking. This pipeline produces diverse, high-quality questions that test complex code understanding in realistic software engineering contexts.
3.1.
Code Processing and Chunking
Repository Selection. We select 12 Python repositories from the SWE-bench dataset to cover diverse domains, as web frameworks, data process-
xarray (987) marshmallow (758)
matplotlib (1017)
sphinx (996)
astropy (997) pylint (992) pytest (804) flask (161) django (1040)
seaborn (405) requests (131) scikit-learn (975)
Figure 2: Distribution of questions of SWE-QA across public repositories. The repositories are 12 open source GitHub repositories that each contains the source code for a popular, widely downloaded PyPI package.
ing, scientific computing, utilities, and varying code complexity. Preliminary analysis of entity density, file structure, and cross-file dependencies informed chunking parameters. Text Segmentation. We used LangChain’s recursive character splitter with Python-aware separators to preserve syntactic boundaries. Chunks of 1000 characters with zero overlap balance context and efficiency, typically keeping functions or classes intact. Separators follow a hierarchy: double newlines, class/function definitions, control structures, single newlines with indentation, and commas. Boundaries preserve syntactic integrity to avoid broken statements. Entity Extraction. We built an entity extraction pipeline based on Abstract Syntax Trees (AST) to systematically analyze Python source code and identify structural and semantic elements. Using Python’s built-in AST parser, the system detects declared entities such as classes, functions, methods, variables, and constants, capturing attributes including type, scope, inferred data type, and defining context. Function and method invocations are recorded separately as called entities, distinguishing definitions from usages. This dual representation enables reconstruction of call graphs, inheritance hierarchies, and data dependencies. Extracted entities are mapped back to their corresponding code chunks through structural and lexical matching, allowing each chunk to reference the entities it declares and calls. The resulting bidirectional mapping links entities and
of in
of
of
of
class
attribute
function function
class tightly class parameter class
and class's function function function function
class function and function function function
class method method
function and
function and function checker variable
class class class function and function
method
class dataarray class function function function reduction
class object function
class class class class function
class class module method
function's class function
decorator class
decorator class method
class function function
and class object argument function method's
entity object functionality function
type entity hierarchy
`eeee` function
and class method responsible defined
and class class method
class function's function
class function's function function function function function attribute attribute function function function
class function function function's function method function method class's class widget userschema
class class used
class class object exception
class class function
class function instance instance function
class class's used function
class class class class class and class class function
class class's method
class entity
class designed
and function function function function function function function context function function function
and and variable attribute attribute
and entity method's method calculation function function's method attribute
class function function function function function function
class context function function attribute function
class function function decorator method function function's
and attribute function
and function method function
and function function function function function method
of
fin
co de
de
ee n `create_tmp_file`
`draw` `create_new_paste`
`date2num` `date_hierarchy`
`cycler` `dataarray.weighted` `cached_etree_parse` `dark_logo` `dataset_to_dataarray` `d`
`assert_identical` `assert_attr`
`as_string` `args.repeat`
`asttype` `asyncclient` `author_schema` `attrgetter`
`docs/_build` `documenterror` `customfieldwithquerysetbutnolimitchoicesto` `downloadfiles` `create_namedtuple_class` `dataclasses.initvar`
`decompose_indexer` `date` `decompose_interp` `_validate_groupby_squeeze` `_template_basename` `day`
`array_notnull_equiv` `assert_representation_allclose`
`aaa` `arange` `acquire_context` `absolute` `autodateformatter`
`before` `axis_direction` `bad_request` `badschema` `baselayout` `bayesian_info_criterion` `baseline` `basename`
`bin_xml_escape` `author` `beta_pdf` `broadcast_dimension_size` `bisect_tests` `book_count`
`truncyear` anyxrefrole
`boomfile` `broadcast_to` `asinhzscalemapping` `cache_control` `anchoredzoomlocator`
`angle_spectrum` `annotate_heatmap`
affinity adminemailhandler circmean
`angle` adminseleniumtestcase
`unitsmapping`
`try_cleanup` `xlim`
`user` `upper` `update_polygon`
`valfmt` `update_sig_from_node` `userinfo` `update_safety_check`
`unique`
`trunc` `warn` `triplot` `transform_non_affine` `transpose` `tree` `transform`
be tw
artificial
`while_used` anchoredauxtransformbox
`write_lockfile` `window_none` `xcorr` alwaysapprovewebprofiledialog
`which` allocation anchoredsizebar adjusted_mutual_info_score admindatewidget `tokenize_module`
addition abstractperson acquire adjusted_rand_score abcdmixin
`zeros` ascii_escaped arraywithnamespace as_variable
apportion aliasinguserserializer appgrouparc appcontext angular_separation
anyindex angleannotation anchoredsizelocator
`validate_raw` `validate.predicate` `validate.url`
`warnings` `validate.email` `validatenocodeerrorconstraint` `variable_type` `wait` `validationerror` `validates_inner` `var` `validate_schema`
spine `match` squeeze `output` redirect
index class `inherited` `put` marshmallow.schem method `copy` logout merge_errors
`make_directive` `reduce` a `norm`
difference
`post` inclusion `posted_at` redefinition `validate` `mock_input`
visitor `append` main duplication `inner` errorstore
abstract
subclassing context
creation removal registration
`visible`
e
return inverse duplicated duplicate definition
sharedcall recursiv
de pe des order reuse im nd ign ple en me
`indexbuilder`
`import_path` `imshow_rgb`
`httpdigestauth` `human_file_size`
`extract`
`fields.raw`
`join_dict_keys`
`legend_elements`
`lettervalues` `duration_iso_string` `fields.ipv4interface` `itrs_to_altaz_mat`
`imshow` `include` `index` `indexdomain` `hist` `infer_datetime_units` `infer_kwarg_from_call`
`isatty` `fields.url` `fields_for_model` `figure_context`
`fields.datetime` `fields.decimal` `extract_zipped_paths`
`field.deserialize` `fff` `fields.boolean.deserialize` `field.serialize`
`homogeneity_completeness_v_measure` category cfmaskcoder
`isattrs` `line_2d_to_3d` `is_serializable`
`line` `list_chunkmanagers` `juggle_axes`
`is_color_like` `is_distribution`
`loggingerror` `login_not_required`
`ensure_deletable` `iter_all_python_module_files` `dummy_return`
`is_title_tuple_type`
`filteredselectmultiple` `file` `filter_like` `find_try_except_wrapper_node` `find_best_app`
`is_inline` `is_method_call`
`is_fully_escaped` `is_file`
`mail_managers` `is_augmented_assign`
`literal_strong` `list` `locationevent._set_inaxes` `load_searchindex`
`is_super` explicitlyindexed percentileinterval
mockmark mockviewuser
`is_old` `is_skipped_package` `get_session`
mockadduser mockdeleteuser
`deserialize_class` `dropaxis`
multiple multireporter
extractday minimumlengthvalidator extraction
mermaidjsprinter
'blue' messageprefixfilter `get_random_string` 'min'
modification multimetric_scorer
navpoint messageidstore
mycookiepolicy mutual naivedatetime modelmultiplechoicefield nanargmax
max_chroma max maxvaluevalidator
nanmedian navigationtoolbar2.forward nanstd navigationtoolbar2.pan nanvar navigationtoolbar2tk navigationtoolbar2wx
`check_func` `check_url_settings` `create_module_file`
`circcorrcoef` `class_name` `clear_script_prefix`
`collect_one_node` `cm` `collecterror`
`delta_to_tick`
median _distributionplotter.plot_univariate_density menuitem mercator merge_setting _apply_indexes_fast
`color_print` `dimensionality` `delete_parameter` `deletedmessageerror`
_finalize `create_icon_axes` `cov` `create_mask` `calc_max_rows_last` `children` `cbook._pformat_subprocess` `callable_or_raise` `celestial_frame_to_wcs` `check_ast_xref_parsing` `check_setting_file_upload_temp_dir`
nancumprod
`colorize_ansi` `correct_roundoff`
noestimator
`density` `create_cleanup_lock` `deque` _get_value_for_keys _parse_array_of_cftime_strings `count`
`contourf` `convolve_models_fft` `convert` `convolve_models`
`cos` `__metadata_request__split` `__array_ufunc__` _strtobool
linkcodeerror adminsplitdatetime linker
numpyrngcontext nouri onetimereceiver
`consumingnofittransformtransformer` `compare_attr` `compat_variable` `confidence_ellipse`
newlinestreamhandler nested nofitindicatorimputer.transform newicrs2
noindextest noinvtransf notransformindicatorimputer permissiondenied norm
quote_schema quote normalize_text
optimization optiongroup orderbylist ra_dec_order pydapdatastore pytest_fixture_setup pytestcachewarning pytestpluginmanager
quotauploadhandler quiverkey
extractisoyear perc
radialdifferential percentile
fake_node fakedirective fakepayload
extractquarter klazz extremefindersimple extractyear
fallback falsechecker outputcodeclassifier passchecker
radialrepresentation propagation
orderedset patched_get_language patchcollection pathgraphingastvisitor pbkdf2passwordhasher
override pairgrid palette palplot paralleltasks passwordresettokengenerator parseerror pass_in_axis
linestyle logical_not literalincludereader's lsof_check
managementutility
latextranslator lack layoutgrid
lexertestmixin label_ranking_loss
lognorm
lessbasicchecker lead least lfplugincollwrapper matern
predictor
m4d manualinterval label_ranking_average_precision_score matchernameadapter mappable markerstyle
library lineartriinterpolator line2d locator_to_legend_entries line_type
polygonselector pluck polynomial1d
readonlypasswordhashfield radioselect
powerstretch power precision_score predict_proba navpoint_navinfo preferred_clock
location ratelimit readtimeout pix2sky_cylindricalperspective rectangleselector receiver
process_index_entry presence prettyprinter processpoolexecutor
projectstate localoutlierfactor `desc_signature_line`
profileschema.dump productionlist luptonasinhzscalestretch
`add_msgid_and_symbol` `dfunc` `add_callback` `default`
`add_node` `add_note` `add_one` `add_uids` `add_artist` `adjust_legend_subtitles` `address` `adjust_bbox` `aliasinguserserializer`
`ancestor`
`size`
check_csp_settings check_file check_header_validity check_memory check_param_validation characterdetailsform check_sts child asttemplateargconstant
`set_options` `split_warnings` `set_yticklabels` `set_zticklabels`
`short_array_repr`
`test_validate`
blitmanager blue boolean astliteral astattributelist astcastexpr astenumerator astropytimelocator astexplicitcast astexplicitspec astfallbackexpr astfunctionparameter
beeswarm binops_overload biweight_midcovariance
astropytimeformatter commonancestor
alias alias_message
childclass childschema childuniqueconstraintproduct
astsizeofexpr base36_to_int astdeclaratormemptr basegeodeticrepresentation
`simmodeltab` `simplejson` `since_created`
iti
on
impleme
relations
in
creation code validation use
of
ntation import
definition
code
class class
class class function
class
and class class implementation method
class function context
read entity class class class class class
function
field method property method method function entity class function function entity object class function method dependent
class class class function
class class model and class function
class function function method method attribute function function
class function's function function
class function
class function function
class method function method method method
class class function method
class function context
class method's
widget function function's function function function function function's function
class function function entity node's function method
class decorator function function function function function function function method's function
mixin class and
field function function's function
class function function
class
class class class class class method function function function
class function
class inheritance
class class function
class dependency
and and function
and class class class class instance function function
class calculation
class projection function function
and method parameter
function function's attribute function function function function function definition function function function function function's function context function
value relationship type method
function class module method function function function
module function function function's
n
definition code
to
betw eenon
of
`dateformat`, attribute use
of
function use
of
call
of
implementatio
method
of
choice decision
class
in
between
instance
of
order
class function method attribute
in
hierarchy hip
class function widget model class designed
class function class's class class class class class function's
class
used
and function method function
class function imported
class, method
class class class node function
class method method method function's function function method
function's field function
field class attribute defined function function method function function function's function function's function function attribute
entity function function function method function function
class method directory
class class method
used function function function
class function
entity variable method function function function's attribute
class class function function function function function function function
and entity class function
parameter and
and entity function attribute method
entity relying attribute
and class class attribute function
and class function function function attribute
class function function's decorator
class class function
entity function matrix function
class class class class and entity class transformation class
class
of
re inheri ntachoicecy pe tan tion at ce ed
re la tio ns hi p
n
io
an the the `stat` `enumerate` cases `returntypenarrowed` `dataclass_with_default_factory.py` issue
instantiating
re
in
are should
the what
the python
using
a
the an
`someclasswithnext`
a
`too-many-ancestors` `super()` `inner_function` `f()` abstract safely class histogram manytomanyfield `transform_content` `final` `set_boxstyle` use and `parse_directive` use functions refactoringchecker saturate rgb_to_husl pix2sky_cobequadsphericalcube parametrize outsideinitmixedin `html_copy_source` `get_co_vertices` `create_test_dataset_attrs` `_get_arguments_inner` `post_load` classchecker 'from' commonpasswordvalidator
the illegalhourerror
the
do did
are
is
two two the some several multiple
certain all there
the
does
it
`last` `app.run` `__all__`
the
`nanmean` `is_system_typevar` multipartparsererror `is_hashable` `astdeclaratorref` `apply_truncate_broadcast_invalid` pix2sky_gnomonic
a3 qsp4
pix2sky_healpix
schema2
radiobuttons pix2sky_tangentialsphericalcube
xarray's mytranslator `taggedjsonserializer`
pix2sky_polyconic create_new_newsfile_if_necessary
yticks they all dictionary
the
the
_query_slice pylint
durationfield mock_cgroup_path outcomeexception pix2sky_hammeraitoff illegalminuteerror httpresponseredirect
there
happ ens
what
how
and and and and and and and and and and and and and and and and intended the keys and and and and and and
any warnings `partialasynccontextmanager` `close` `options` `np.linspace` `log_namelist` `is_deleted_symbol` `save_traceback` `asttypeusing`
`regen` `checkbuttons`
`cast` `blogonlyschema`
`author` `attributeerror` `add_fancy_patch_around` `accept` `readable` `release_zoom` `_unary_ufunc` `deserialize` `get_module_filename` `from_timestamp` `formfield_overrides`
`fields.tuple` `extract_changelog_entries_for` `field_class`
`eventadmin` `encode_string_array`
`draw_circles` `default_key_func` `inherit_from_std_ex`
data effect specific triggers
`dateformat` `containsnoneof` `condition` `compile_matchers` `combining_coroutine1`
`dropshadowfilter` `validationerror`
are
`validate` `validate.length.max` `userrelativeurlschema` `useless-parent-delegation` `test_field_deserialization_with_user_validators` `unwrap` `testattributes` `_get_ini_as_rst`
`wrapper` luv_to_lch methods mock nansum given nfplugin nodecount options ordereddict override_environ pix2sky_cylindricalequalarea pix2sky pix2sky_zenithalequalarea
customwriterform aaaexception integers ansi arguments choices coordinatematchresult classes
custom plot data_key deletedmessageerror
deprecated diamond donothing dummyentrypoint format inverse instances _sexagesimal section `_validate_colour_support` _parse_args
try transformations streaminghttpresponse `watch_for_template_changes` sphinxrole simplifiable set_xbound save_icon inclusion rgb_to_hex readme.rst purge_module
static with warning validation user_data
unit union nested implementations functions
tests
abstract
use
two the
toif
when
types does configuration
the
django's setting using
wcsaxes.contourf sphinx's
pytest_configure pytest_configure's pytest_exception_interact pytester.parseconfigure pytester.runpython_c's schema.dump() silentattrclass's pytest_collect_directory my_function's `assert_duckarray_allclose` `get_pkg_data_fileobj` `is_node_in_type_annotation_context`
`isbuiltin` `isclose`
`isstaticmethod` `monkeypatch.chdir(path)` `nan_to_num` pylinter.add_message paralleltasks.join wcsaxes.scatter_coord wcs.deepcopy wcsaxes.format_coord wcsaxes.get_coords_overlay wcsaxes.grid sphinx.add_domain sphinx.add_post_transform sphinxwarning temppathfactory.mktemp silentgetitemclass's pytester.runpython navigationtoolbar2 get_import_prefixes_from_env
extending
featureunion.transform
django dateformatschema.activation_date class assigning `utils.is_collection`
frame1
get_image_extension gettempdir highlevelwcsmixin importmode.importlib's importorskip iterate_nested maskableshapedlikendarray monkeypatch.delenv monkeypatch.delitem cachingfilemanager.acquire_context `np.linspace` `self.dump_only` `stash.setdefault` `train_test_split` `truncsecond` `utils.from_timestamp`
is
does
a
a
in
in
a
a
in
django's
django
wcsaxes.set_xlabel wcsaxes.set_ylabel xarray xarray's wcsaxes.get_xlabel() wcsaxes.get_ylabel() wcsaxes.get_tightbbox wcs.sub using
xarray's weighted ticks ticklabels static
subscriptable subclasses
find_prefixed filter_whitespace cicharfield axislabels `ispartial` abstract inline
pandasmultiindex suspiciousoperation
sphinxposttransform spectralcoordinate simpledataobjectone python's
python
the
matplotlib's matplotlib arrows python's
linterstats.get_global_message_count `broadcast_to` `axvspan` `assert_duckarray_equal` `assert_allclose_dense_sparse` `desc_sig_literal_number` marshmallow linterstats.get_module_message_count homogeneouslist's sphinx gridsearchcv flask's featureunion facetgrid.map_dataframe dotbackend.generate featureunion.fit `resolve_collection_argument` `pytester.runpytest_inprocess` `pylinter.set_current_module` `notrans.fit` `config.cwd_relative_nodeid` pytest_runtest_logfinish `messagedefinitionstore.get_msg_display_string` math.isclose pytest_internalerror pytest.raises pytest.mark.parametrize pylint's polyfit percentstylemultiline.format np.interp minibatchkmeans
instances unmanaged
a
the
django's
matplotlib's
is
the
the
marshmallow's pytest's flask wcsaxes matplotlib
are
a
the
can
do
the
calibratedclassifiercv `send_from_directory` `sphinx.util.inspect.stringify_signature` `unique_markers` `url_params_from_lookup_dict` astropy's `run_using_a_configuration_file` cross_val_score datatree.match `is_builtin_classmethod_like` `encodedgroups` `evaluate_xfail_marks()` `facetgrid.set_axis_labels` `filled` `fixedformatter` `get_config_dir_path` `get_readable_fileobj` `get_scope_node` captureio `isinstance` `make_scorer` `monkeypatch.setitem` pytester.getitem pytester.popen pytestunhandledthreadexceptionwarning ridge setting spectralclustering pytest_runtest_call sphinx.util.inspect sphinxerror sphinxparallelerror pandasmultiindex pytest_runtest_makereport matplotlib.pyplot.subplot
Code Features
pytester's
duckarray2 model2dkernel sphinx-build selectkbest's
fk4 customtd2 `testclass` `rcdefaults`
histeqstretch graphicscontextbase
memoryfileuploadhandler's
inversesip
raise
when
where should
will
can
do
Figure 3: Ratios of code chunks showing the presence of specific programming constructs (loops, conditions, functions, classes, imports, async operations, and exceptions).
Question Generation Process. We used MetaLlama/Llama-3.2-3B-Instruct, tuned for creative but precise output, with prompts specifying context, question type, and multi-hop reasoning requirements. An iterative refinement checked clarity and multi-hop validity. Semantic analysis ensured structural diversity, varied vocabulary, and appropriate complexity.
of
between
why
correctly correctly preserve truncate handle
Question and Answer Generation
of
of
are
correctly and rely behave rely use correctly affect inherit correctly raise ensure
Chunk Sampling. Candidate pairs for DC and triplets for IE were enumerated across repositories and validated for genuine multi-hop relationships. Selection enforces diversity: limiting questions per entity or entity pair, balancing repositories by size and complexity, and ensuring a mix of simple and complex patterns. We capped each repository at 600 questions per category and applied quality thresholds for relationship strength and code complexity.
of
is
handle enforce rely a1
Multi-Hop Taxonomy. Inspired by the HotpotQA dataset, we have defined two multi-hop question types. Declaration-and-Call (DC) questions connect an entity’s definition in one chunk with its usage in another, requiring reasoning about parameters, return values, state changes, or error handling. Interacting Entity (IE) questions involve three chunks where two entities interact, demanding analysis of data flow, execution order, shared state, or collaborative behaviors.
of
use
80
Question Categories and Generation
of
does
correctly change handle interact execute consider create always modify have register and serve ensure implementation handle have have
Percentage of Questions (%)
of
the
interaction inherit return correctly correctly
60
of
ct ra
the np.isnan(pixel[:, rely correctly `add_role` `add_autodocumenter` raise usage have create dependency handle property
40
a
te in
`html_theme_options.sidebar_hide_name` `django_allow_async_unsafe`
20
it
tio n
the
0
percentstylemultiline's pytest_ignore_collect
functions
truncdate timedelta
imports
3.3.
ac
variable.unstack
conditions
a of
the
recursive redefined_by_decorator register_cleanup_lock_removal reindex sphinx.disconnect random_walk similaritieschecker simpleimputer singleentryproject sklearn slicedlowlevelwcs samphubserver's query quantity rstparser's run_compile safe_median saferepr sampclient samphubproxy randint pytest_make_collect_report pytestassertrewritewarning pytester's pytestpluginmanager's python quadraticdiscriminantanalysis spectralquantity sphericalcoslatdifferential self pydapdatastore series_reduce shared_axis_remover sharedaxesleveltests shield shift select_step_hour moffat2dkernel metaclass metaregressor's method min mineral minimaltransformer minmaxinterval modified messagemiddleware messagestyle seasonresampler scriptinfo.load_app sciencestate showversion sphinx.setup_extension sphinxdirective sphinxfileoutput sphinxfilesystemloader sphinxrenderer sphinxsmartquotes sphinxtestapp.cleanup splineinterpolatefitter selftrainingclassifier save_results scanner.reject scatter sphericalcircle monkeypatch rstparser relationmodel roleclasses restify release renamemodel render_human_readable_nbytes replace report representationmapping reprfuncargs requestfactory residplot monkeypatch.setattr monkeypatch.setenv monkeypatch.undo check_setting_language_code checker_class checkertestcase checkmessage categoricalnb child choice circlepolygon classdiagram classicalmds check_pos_label_consistency classifiermixin clip binary_operation atan authenticationform autodoc autodoc_attrgetter autoscale axline band basicchecker classproperty check_interactive_exception check_duplicates charfield versionedpackage userschema.age userschema.email uuidfield v_measure_score variable variablecommentpicker vectorizedindexer version logcapturehandler wcs.dropaxis wcsaxes.format_coord wcsaxes.grid weightedmetaregressor's withnode writer checkregistry changelog clustergrid bincount booleancoder compositestretch compress astype conditionalgetmiddleware's connectionpatch connectionrouter consolidation constantkernel construct_instance coordinate complementnb coordinatetransformindex `validate_html_static_path` `validate_intersphinx_mapping` `validate_parameter_constraints` `validate_physical_types` `validate_schema` `validate` `validator` `var` `variable` `validate_dataarray_coords` `verify_description_mode` compact_paragraph comb collectreport broadtryclausechecker button calculate_dimensions customerror count_nonzero creation csd cumsum custom_frame_to_wcs_mappings custom_ucd_coord_meta_mapping cot customusercreationform's d2_absolute_error_score d2_log_loss_score dataarrayaggregations dataset.map datasetaggregations compute_module_name code collection biweight_scale menuselection messageencoder polartransform pathawarehookproxy percentformatter's permwrapper pinv pipeline plot_tree plotdata median plotter polynomial2d pprint presence primer primercommand private profile protocol onevsrestclassifier nodematcher patch_docutils md5passwordhasher matcher maskedcolumn multioutputclassifier multiplechoicefield multiplelocator naivebayes manytomanyrel logisticregression logxml lookupkey lrap_score main make_glossary_term mangle_signature manualpagetranslator nameresolveerror marshmallow.schema marshmallow.schema.schema nonconsumingclassifier nonuniformimage normalize_choices notchecker urlize usage user_cache_dir test_runner split statelessestimator's step storage_class stringformatchecker subplot tbfilter templateview userschema testfilter testframe testgroup testinterval testreduce text textwriter themeerror unsetmetadatapassederror `view` unparse universalcontainer nullreporter nulltimekeeper nystroem splinesmoothingfitter order orthogonalmatchingpursuit outliermixin outputline overlappingexceptionschecker packagetolint parallel parkinglot typingalias ticklabels timefield token toolmanager top_k_accuracy_score transform_contour_set_inplace triangulation typechecker third_walk underline unmangle `check_csrf_cookie_secure` `baseradecframe` `center` `get_namespace` `get_node_line` `get_node_source` `get_params` `get_actions` `full` `func4` `func` `gaussian1d` `gen_even_slices` `generate_coords` `generatedfield` `genericprefetch` `get_parser` `get_auth_from_url` `get_backend` `get_best_relpath` `get_child_arguments` `get_connection` `get_converter` `get_db` `get_msgid` `get_metadata_routing` `get_layout_engine` `get_lambda` `set_session` `set_style` `message_flashed` `set_theme` `set_trace` `set_xscale` `setdiff1d` `setup` `setupmodule` `setuptestfs` `get_default_password_validators` `sharedaxesleveltests` `get_downloaded_sites` `get_encoding_from_headers` `get_feature_names_out` `get_fignums` `get_font` `get_fontawesome` `get_from_config` `get_functional_test_files_from_directory` `get_image_size` `get_diadefs` `get_installed` `good_cm_finally` `getgroup` `getitems` `get_template_attribute` `get_unpatched` `get_xticks` `get` `getbasetemp` `getfixturemarker` `extractyear` `exception_str` `exclude` `ceil` `exp` `explicitlyindexed` `extract_zipped_paths` `extractisoweekday` `extractquarter` `extractweek` `excepthook` `fake_home` `fakeframe` `fakeicrs` `expect` `set_requests` `full_like` `get_reloader` `getlogger` `getslots` `getstatement` `gettext_lazy` `getvalue` `global_toctree_for_doc` `getfuncargnames` `good_cm_no_cleanup` `good_cm_yield_none` `gradient` `get_resolver` `grid_to_graph` `group` `guess_filename` `guess_mimetype` `get_stderr_fileno` `get_password_validators` `get_path` `get_pc` `get_pkg_data_contents` `get_printer_for_filetype` `get_random_secret_key` `gridsearchcv` `featureunion.fit` `set_output` `set_aspect` `unquote_unreserved` `unrelated` `to_netcdf` `thing` `three_item_iterable` `title` `to_dataset` `to_dict` `to_iris` `to_key_val_list` `to_native_string` `spanselector` `to_numpy` `to_rgba_array` `to_utf8` `tokenwrapper.line` `trace_view` `trace` `tracemalloc_message` `transform_non_affine` `schema_editor` `unpack_for_decoding` `unique` `unique_inverse` `unique_all` `spy_factory` `sqrt` `textwriter` `stack` `stackplot` `stream.close()` `stream_template` `stream_with_context` `streamplot` `strindex` `safe_isclass` `strip_escape_sequences` `truncate` `try_argcomplete` `try_import` `try_makedirs` `type` `typing.any` `transform_path` `underline` `uninstall_repl_displayhook` `undecorate` `safeformat` `sample_y` `s` `safe_cast_to_index` `simple_norm` `should_ignore` `shuffle` `sigmaclip` `sigmaclippedstats` `signature_from_ast` `signature_from_str` `runtestprotocol` `should_bypass_proxies` `skip_on_field_errors` `skip` `skycoord.skyoffset_frame` `solver` `source.strip()` `source` `space_indentation` `set_temp_cache` `serve_application` `sessiontests` `simplenorm` `set_cookie` `runitem` `run_checks` `save_estimators` `sched_getaffinity` `schema.load` `safe_exists` `score_samples` `search_around_3d` `search_around_sky` `search` `section` `self.many` `run_with_reloader` `sep` `rmtree` `response_class` `response` `returns_something` `reverse` `rfc1123_to_epoch` `serialize_context_as` `roll` `round` `separate_metadata` `fetch_20newsgroups_vectorized_fxt` `ensure_file` `inherited_coords_repr` `init_app` `init_console` `init_db` `hist` `handle_exception` `handler` `has_change_permission` `has_even_axis` `hashable` `hasmemoized` `help_message` `highlight_block` `husl_palette` `identical` `ignore_event_outside` `is_iterable_but_not_string` `infer_kwarg_from_call` `index_errors` `indent` `include` `make_app` `make_auth_header` `make_axes_area_auto_adjustable` `make_entry` `make_hashable` `make_id` `make_link_node` `is_suppressed_warning` `make_regression` `make_response` `is_clusterer` `make_s_curve` `makeconftest` `makefile` `mangle_test_address` `margins` `infer_all` `imageadapter` `imagehdu` `import_by_name` `import_object` `in_type_checking_block` `make_sparse_random_data` `is_collection` `is_default_argument` `is_empty_str_literal` `is_builtin_class_method` `is_builtin_object` `is_bytes_dtype` `is_classifier` `is_aware` `make.run_generic_build` `is_attr_protected` `is_assign_name_annotated_with` `is_error` `is_exception` `is_fs_root` `is_it_a_good_day` `is_classmethod_descriptor` `is_none` `is_postponed_evaluation_enabled` `is_regressor` `is_separable` `is_serializable` `is_async_function` `is_smartquotable` `is_supported_builder` `is_allowed_version` `inline_genitems` `inline_runsource` `inline` `inspect.signature` `int32` `inverse` `invert_yaxis` `ipaddress` `is_super` `fetch_20newsgroups` `make_lupton_rgb` `merge_doctrees` `float_power` `floatfield` `flushto` `fooserializer` `forbid_multi_line_headers` `format` `formatstrformatter` `fspath` `fieldsonformonlyadmin` `ffmpegwriter` `fields.constant` `fields.int` `fields.ip` `fields.raw` `fields.str` `fields.url()` `fields` `halfbinomialloss` `filename` `fill_between` `fillna` `find_pending_xref_condition` `flask.redirect` `flash` `fix` `dump_to_store` `dumped` `ecdf` `empirical_covariance` `enable` `encodedstringcoder` `fetch_kddcup99_fxt` `ensuredir` `epubelementtree` `filter_like` `escape_abbr` `euclidean_distances` `evaluate_skip_marks` `exc_type` `except*` `flask.session` `findfont` `findsource` `fit_predict` `fit_transform` `fits2bitmap` `escape` `filter_traceback` `finalise_release_date` `mainloop` `json` `jsonify` `kernelridge.fit` `kernelridge` `key_field` `kw_only` `label` `labelbinarizer` `maxnlocator` `isscalar` `markgenerator` `maskedcolumn` `max_abs_diff` `mark_safe` `maybe_wrap_pytest_function_for_tracing` `mdates.concisedateformatter` `mean_squared_error` `mean` `median` `memorycachedarray` `marshmallow.validate` `merge_setting` `isotonicregression` `isomorphic` `lazilyindexedarray` `len` `less` `library.filter` `linearclassifiermixin` `lineardiscriminantanalysis` `lintmoduleoutputupdate` `list_engines` `load_config_dict_from_file` `last` `isort` `log_to_file` `logging-not-lazy` `logical_or` `lpad` `lstsq` `join_dict_keys` `is_typing_member` `is_unity` `isbuiltin` `isnat` `logger.log_to_file` `split_explicit_title` `rgb_to_hsv` `spectral_embedding` _get_pylint_home() _get_logger _freedman_diaconis_bins _enumformatter.preamble_lookup _digits_dataset _categoricalplotter _assert_internal_invariants _add_colorbar _categoricalplotter._adjust_cat_axis `radians` `radioselect` `popcall` _get_report_choice `pd_timedelta_to_float` _get_safe_url `_categoricalplotter` `_convert_to_numpy` `sphinxfileoutput` _is_numeric _infer_concat_order_from_coords `pending_logging` `persist` `physical_lines_for_line` `repr` `represent_as` `reprhtmlmixin` `request` `requestcontext` `requests` `reset_margins` `register_colormap` `raisesgroup` `randint` `randomtreesembedding` `re_match_lines_random` `read` `rec` `recwarn` `colorize` `color_palette` `collect_unraisable` `collect_one_node` `repr_traceback_entry` `relpath` `repr_args` `rep` `pipeline.set_output` `pipeline` `plot_angle` `poly3dcollection` `raise_build_error` `pos` `positivesmallintegerfield` `pre_release` `predicate` `_check_argument` `prefix` `preprocess` `prev` `repr_callable` `remove` `removedinsphinx90warning` `rename_dims` `render_template_string` `render` `reorder_items` `prepare_import` `_check_initialpaths_for_relpath` `_check_response_method` `pytest_pycollect_makemodule` `process_author` `protected_access` `pylinter.get_ast` `pytest.skip` `pytest_collect_file` `pytest_exception_interact` `pytest_fixture_setup` `pytest_ignore_collect` `pytest_pycollect_makeitem` `primarykeyuuidmodel` `pytest_runtest_setup` `pytest_sessionfinish` `pytester.chdir()` `pytester.mkdir` `pytestfdwarning` `r2_score` `dtype` _validate_usepdb_cls _test_environ_pythonpath _setoutputmixin _resolve_intervals_1dplot _pytest.pathlib _proj_transform_vec_clip `==` `_gen_file_data` `_format_lines` `_fix` `_fit_context` `_ensure_padded_year` `_do_configure` `_deserialize` `_cpu_count` `_arrayize_vectorized_indexer` `_combine_indexers` `_check_shape_tile_ids` `_check_sample_weight` `.copy()` `.pathpatch_2d_to_3d` `@deprecate_positional_args` `_check_partial_fit_first_call` `_annotated_unpack_infer` `_add_rcfile_default_pylintrc` `__sklearn_tags__` `__repr__` `__new__` `__class__` `__add__` `clsunsubscriptable` `cm` `mktemp` `next` `mockrequest` `modelboundingbox` `modelwithdatabasedefault` `modify_sys_path` `module_available` `move_legend` `move` `mtime` `parametrize` `open_files` `open` `ordered_set_union` `orient_indices` `pack` `pairwise_distances` `palplot` `param` `mkpydir` `missing_` `minorticks_on` `miniuserschema` `my_function` `myattr` `mycontextmanager` `myerror` `myform` `myschema.load` `name2` `multioutputregressor` `names` `open_datatree` `nanmean` `navigationtoolbar2tk` `navigationtoolbar2wx` `nested_parse_to_nodes` `nested` `next_lower` `mock` `metadatarequest.consumes` `metatransformer.fit` `method1` `nativeenumcoder` `parse_args` `parse_data_uri` `parse_header_links` `stripplot` `supported_dtypes` `supports_membership_test` `suppress_logging` `swarmplot` `sym` `syspathinsert` `table.set_separator` `taggeditemform` `teardown` `strip_units` `template_changed` `test_client` `test_required_message_can_be_changed` `testdir.makefile` `testform` `text_2d_to_3d` `textarea` `squared` `spawn_pytest` `teecaptureio` `multireporter` `onevsrestclassifier` `on_rm_rf_error` `parse_num` `parse_xref_object` `pass_fail_on_config_to_color_reporter` `patch.object` `pathawarehookproxy` `numdictcontext` `nonconsumingclassifier` `nonsingular` `normalize` `onetoonefeaturemixin` `not_equal` `notnull` `notrans` `null_session_class` `resp.content` `obsgeo_to_frame` `offset_by` `on_changed` `on_close` `on_commit` `notify_exception` `cls` `multipolygon` `release` `connection` `colour` `config_context` `config.add` `config._ensure_unconfigure` `concatenate` `concatenate_representations` `compute` `namedtuple` `commonpath` `conj` `check_memory` `check_isinstance` `check_inplace_ensure_writeable` `consider_conftest` `check_header_validity` `consider_module` `contains_cftime_datetimes` `register_checker` `_get_members_to_document` `pav2pv` `register_dataarray_accessor` `register_message_definition` `clear` `clear_checkbox_id` `cleanup` `clean_astext` `class_registry` `class_registry.get_class` `convert_units` `context` `context_manager` `consider_preparse` `check_get_feature_names_out_error` `compress` `check_cache_location_not_exposed` `reindex_variables` `reindex_like` `regressormixin` `regplot` `register_reporter` `cftime` `check_allowed_hosts` `check_file` `check_confval_types` `check_consistent_length` `check_default_cache_is_configured` `check_csrf_middleware` `circcorrcoef` `check_content_type_nosniff` genericipaddressfield `positivebigintegerfield` `sqrt` built-in `check_referrer_policy` `booleanfield` selectmultiple related_name proxy include
loops
3.2.
er
name2 wcs.to_header's
`_deprecate_positio `decision_function` `sigma_clip` `dummydomainsco `dummyregressor.s `dump_only` nal_args` httpadapter `set_config` `make_filename_fro ntainer` `set_params` core` `basechecker` `post_load` `predict` `code.from_functio m_project` `assert_outcomes` `pass_original` `biweight_scale` `pop` setup n` `rewrite` `runtest` `connect` `register_messages `register_plugins` relationship `partial_fit` `assert_chunks_equ `collect_by_name` _from_checker` `orderedset` `only` `parser` `relative_luminance al` `addfinalizer` `min` dummyclassifier's dot increment ` iterativeimputer's `authorschema` `where` `upper` histogram `decimal` `debug_sql` `remove_duplicates` `add` writetootherrouter call count dataset.weighted `getfslineno` `fancyarrowpatch` `transform` `saferepr` `signature` `result` `split` `init` `initialize` `ignore_warnings` `fields.decimal` `super_len` explicit_indexing_ada `sum` emphasizedliteral environbuilder abstract `getdoc` pter auxtransformbox definition `find_best_app` `fit` `is_file` `floor_divide` dataarray `ipaddress.ip_interfac `formattedexcinfo` `install_stubs` `dump` `filterwarnings` `lower` e` `close` `inverse_transform` `clip`
np_cov_ind
in t
`add_enabled_equivalencies`
`print_red` `rotate_token` `self.allow_none` `self.birthtime` `self.labels_text` `self.metadata`
`self.name` `self.parent` `self.set_class`
`register` `quantity_support` `quantityinfobase`
`quantity_asanyarray`
`prev` `print_contents` `pointclass.x`
`read_po` `read_svg_depth` `recordedhookcall` `register_lookup`
`emit_edge` `getheader`
`sep` `proj_plane_pixel_area` `remove` `print_full_documentation`
`polynomialmodel` `polyfit`
`possible_exc_types` `polyval`
`get_translation`
`get_verifier` `get_wcslib_cfg`
`insert`
`raise_last_exception` `quote` `read_magic_number_from_file` `raises`
`getmodulecol`
`greet` `grid_locator2`
`groupby` `guess_json_utf`
`post_load` `predict` `prefix` `regplot` `primary_contact` `emit_pragma_representer` `empty_ip`
`lines` `label` `labels` `lassoselector` `internet_on`
`get_stderr_fileno` `get` `get_system_username` `get_symbol` `get_template_attribute`
`ipv6_exploded_string` `get_xy`
`instance` `getstatement` `getstatementrange`
captureresult
cast_to_int_if_safe cast cataloginfo catalogrepository
`length_class` `http_server` capturequeriescontext
dbdefaultsfunction derivedfromunknowngrandparent
choicefield capturemanager bubblechart bytes_to_char call_fixture_func callable callback check_setting_languages_bidi
`test` `text2d` `set_environ` `set_cookie_data` `textbox.on_submit`
`sip_foc2pix` akaike_info_criterion
`skip` `shrunkcovariance`
`set_filter` `set_hls_values` `set_missing_parameters`
`split_pulls` `set_xscale` `tabular_model`
`sigmaclip` `signature_from_ast`
dummyterminalwriter dummyregressor duckarraymodule dtype_info_name dt.datetime duration_string int_list_validator inputparametererror `release_date` `get_script_name` `relationship` capstyle custommodelchoiceiterator custompermissionsusermanager customprovider customset customuseradmin customuserchangeform customwidget chunkedencodingerror cut_lines cylindrical cylindricaldifferential darkgreen data_key dataarrayrolling dataset dataset.to_netcdf bar asttemplateparamnontype astthisliteral astwalker autodetector average_precision_score awaredatetime custommapping customwriterform desaturate davies-bouldin commonpath comparison compat_dict_union composition compoundkernel condition deep datatree.load datetimelist datetimepickershortcutsseleniumtests
copy_asset_file
`slice_slice_by_array`
chi2_kernel check check_alignment check_alternative_lrap_implementation
`smooth2d` `someval` `spam` `specgram` `spherical_offsets_to`
common customdefaultredirecturlloginview
decimal decision decorator's datasetrolling default_indexes defaultjsonprovider deg2rad densify deprecation derivedbook axvline bandadmin asttemplateparamconstrainedtypewithinit
`to_cartesian` `to_geodetic`
box2dkernel astnoexceptspec astpackexpansionexpr astrequiresclause astropydatetimeleapsecondwarning
`stopupload`
`to_rgb` `str_dump_only`
copystat conditional counternode countvectorizer's cplsuccessivelineslimits
colourise circular code codetypecount collect_test_modules
`split` `spouse` `stack_negative` `start_of_file`
`start` `timeout` `timestamp_as_unit`
`stripped_lines`
`findobj` `findsource` `fit_deriv`
`set_axis_labels` `showfixtures` `sky2pix_slantzenithalperspective`
`get_scope_node`
`table_lines_from_stats` `stop`
extendsnode `fourthgoodclass`
`sub` `subst` `sum` `suptitle` callequal capturefixture._start
configerror `timedelta_to_microseconds` correct
config.getoption csp_override cubehelix_palette custom_stokes_symbol_mapping cursor
`ticks` colour covariance connection contactform contourlabeler coordinatetransformindexingadapter conversion
`store` assignment `str_regular` `stringnotcollectionerror`
`setitem` `shade` `shift` `show_versions` `set_coords` `showbase` `text_to_rgba` `tempdir` `temporary_verbosity` `test_context_test` `test_email_custom_message` `test_field_deserialization_with_user_validators`
`reindex_like` `serialize` `get_routing_for_object`
`fit` `float_column_formats` `flatten`
`get_connection` `get_exception_info`
`format_doc` `formfortestingisvalid` `make_bench_data` `gaussian_mixture` `generator_factory` `get_all_symbols` `get_attribute` `get_authors` `get_config_dir_path`
`raise_error` `pytest_collectstart`
`get_quote` `get_pull_request` `get_public_names`
`empty` `end` `findfont` `entry` `error_messages`
`exploded` `empty_ipinterface`
`run` `runpdb` `saved_fd` `schema_editor`
`round_hour`
`evaluate_xfail_marks` `error`
`get_new_headers` `get_minimal_setting`
`evaluate` `expect_three` `extract_codec_from_bom`
`run_memleak_test` `round`
`get_is_old` `get_issues` `get_log_level_for_setting` `get_level`
`scaled` `project.path2doc` `protocol_callback` `season_to_month_tuple`
`get_global_message_count` `get_from_dict` `get_image_filename_for_language`
`get_milestone_id` `pytestunhandledthreadexceptionwarning` `get_files` `pytest_sessionfinish` `pytestunraisableexceptionwarning`
a
`add` `distancetolonlat` `divider.new_locator`
a
using
nce
despine drawtree dt.time duckarraywrapper dummy_function dummyapplication dummychunkmanager def_unit datatree.to_dataset dotbackend datatreeaggregations decimal.decimal decisiontreeclassifier decisiontreeregressor decisiontreeregressor.fit deep_align dummyscorer default defaultvalue definitionparser definitionparser's dateenum docutils_namespace distribution locallylinearembedding locatordms inline_runsource index indexcallable indexing inheritancegraph inheritdict inheritexit geocentricmeanecliptic inline_variable_array_repr invalidjsonerror invalidurl inverse inverse_permutation isnat iterativeimputer download_file det_curve diadefgenerator diamondsubsubcounter delete locally deprecated_renamed_argument design filter_indexes_from_coords filteredmapping fit_transform fit_wcs_from_points `update_differentials_to_match` `asyncmanagermixin` `assert_sparse_equal` `assert_vector_equal` `assert_writeable` `assert` fields `assertionrewritinghook.exec_module` `assertoutcome` `assigned_bool` `astexpression` `assert_node` `atomic` `attempt_symlink_to` `attrs` `augment_sys_path` `authorschema.last` `assertionrewritinghook` fastfilescompleter factory fact flask flask.session flatten footnotetext frozenestimator's fixturemanager functionassigner functionchecker galactocentric gaussianfilter generate_autosummary_docs generictestreporter facetgrid.map_dataframe dummytemplateloader dummytransformer epubbuilder's errorbarcontainer estimatorcheckfailedwarning exceptionraiserefvisitor expsinesquared extravalidationformmixin deserialization `auto_assign_coord_positions` linttestupdate lineartriinterpolator assertionrewriter assertionrewritinghook applicationerror assertionstate astdeclaratornameparam astmacro astnestednameelement astpostfixmember astropyuserwarning astropywarning assert_time_equal asttemplateintroductionparameter `wrap` `wrapper` `wraps` `write_mo` `write_output` `write_sep` `write_thumbnail` `write` datatree.assign `yearlocator` `x` assert_quantities_allclose arraywithoutbroadcastto argumentpreprocessingerror `with_metaclass` `wordwrap` `user_data` `update` `url_for` `url_params_from_lookup_dict` `urldefragauth` `usageerror` `user.age` `wrap_angle_at` `userdict` `userexcludeschema` `userschema` `utils.decoding_stream` `validate.equal` `validate.length.equal` `validate_basetemp` `validate_data` assertionrewritinghook's arc argmax `zaxis_inverted` lintmoduleoutputupdate `zip_longest` accuracy_score get_test_data getattr getfuncargnames gettextrenderer golden_spiral_grid googledocstring grad gradient gradient_bar grandparentleft get_random_secret_key gzipmiddleware's jointgrid jsonreporter kernelridge kfold kneighborsclassifier kneighborsclassifier's label_from_attrs labelencoder lexer's line_search latexbuilder get_encoding_from_headers get_auth_token get_argument_from_call add_cell address_in_network anchorcheckparser html5translator handlerline2d hdulist heatmapper heliocentricmeanecliptic hiddeninput highlevelwcswrapper highlightlanguagevisitor handler htmlthemefactory httperror httpresponse humanerror hyperlinkavailabilitychecker identity imagefield implicittoexplicitindexingadapter getrawcode get_and_validate_output_file abbreviation `asarray` `area` `argmax` `decode_cf` `decoding_stream` `dedent` `default_order` `create_files` `copy()` `copy_example` `copyfile` `correct_copyright_year` `dec` `create_app` `create_figure` `defaultfillvaluecoder` `create_jinja_environment` `create_stubs` `create_terminal_writer` `create_test_datatree` `create_vlen_dtype` `cross_val_predict` `ctime` `custom_ctype_to_ucd_mapping` `create_coords` `custom_ucd_coord_meta_mapping` `datetimefield` `_wrapcall` `_write_appconfig` `_write_pyc` `_write_with_fallback` `abcd` `abort` `category` `absolute` `absolutepath` `add_attrs` `add_config_value` `add_event` `add_latex_package` `add_msgid_and_symbol` `add_task` `customdescriptorfield` `customtypedfield` `cwd` `dataarray.load()` `dataarrayweighted` `dataset.chunk` `dataset_update_method` `datetime.timedelta` `do_thing` `_windowsconsoleio_workaround` `discover` `display_chunk` `detrend` `diff_array_repr` `diff` `dirname` `disable` `check_templates` `check_pickling_recovery` `check_security_middleware` `check_session_cookie_httponly` `check_session_cookie_secure` `deserialize_class` `check_setting_file_upload_temp_dir` `check_sts_preload` `check_sts` `check_non_negative` `check_transformer_get_feature_names_out_pandas` `check_url_settings` `check_vlen_dtype` `check_xframe_deny` `check_xframe_options_middleware` `check` `childuniqueconstraintproduct` `check_setting_language_code` `desc` `desc_annotation` `cookiejar_from_dict` `display_rgb` `display` `distplot` `divide` `do_something` `disconnect` `docstringify` `docstringparameterchecker` `dotbase` `draw_error_band` `draw` `dropna` `dropshorterlonghelpformatter` `dstack` `deque` `defaultjsonprovider` `deindent` `delaxes` `delete_masked_points` `delete` `demo_locatable_axes_hard` `densitymixin` `depthshade` `dispatchingjinjaloader` `_version_predates` `abs` `applysourceworkaround` `cachingfilemanager` `call` `callback` `canon_path` `caseinsensitivedict` `catalogrepository` `basereport.toterminal` `autosummary_dummy_inherited_module` `autosummary_toc` `await` `cached_etree_parse` `backends_dict_from_pkg` `baseconstraint` `baserenderer` `approx` `basereporter` `beeswarm` `bigautofield` `binaryfield` `binomial` `biweight_location` `bar` `cache` `bool` `bytes_to_char` `args_with_annotation` `args` `argsort` `array_notnull_equiv` `array_repr` `as_float_array` `autoscale` `aschema` `assert_allclose_dense_sparse` `assert_colors_equal` `assert_coordinate_consistent` `assert_date_equal` `assert_identical` `assert_never` `assert_no_warnings` `bytes` `box` `broadcast_compat_data` `build_engines` `build_main` `build_plot_signature` `build` `bump_prefix` `blogonlyschema` `blogschemaexclude` `_safe_indexing` `_process_docstring` `_make_fontconfig_parser` `_netcdf4_create_group` `_network` `_parse_confdir` `_parse_content_type_header` `_parse_doctreedir` `_parse_group_and_groupers` `_parse` `all_numeric` `addoption` `adjust_legend_subtitles` `adminsplitdatetime` `adminuuidinputwidget` `alias` `align_nd_chunks` `add_transform` `allow_none` `allowsnullgfk` `altaz_to_hadec_mat` `and_expr` `app_context` `apply_source_workaround` `_unstablearchmixin` `warn` `_load_theme_conf` `_is_contiguous` `_pseudo_parse_arglist` `_read_pyc` `_readline_workaround` `_remove_old_files` `_repr_png_` `_reset_epoch_test_example` `_rget_with_confmod` `_run_pylint_config` `_patch_colormap_display` `_session` `_setoutputmixin` `_should_cftime_be_used` `_splitstrip` `_strip_resource_warnings` `_synth_regression_sparse_dataset` `_threshold_scores_to_class_labels` `_token_type` `_load_theme_with_ancestors` `_get_pdata_path` `_get_sections` `_getpytestargs` `_import_module_using_spec` `_ipython_key_completions_` `_liveloggingstreamhandler` `choices_text` `warn_explicit_for` `versionrequirementerror` rangeindex rationalquadratic
classes
in
`age` `allow_none_field` `allow_nan` `_inherited_dataset` `_check_functional_tests_structure` `alogout` `_check_dependencies` `__str__` `_add_line` `_check_optimize_result` `_generate_missing_indicator_cases` `_distribute_candies_to_child` `_compare_approx`
aa
there
Code Features in Dataset async
to
`_node` `add_attrs` `discretize_oversample_2d` `_union1d` `discretize_model`
overriding designed designed f_classif use
interaction
and dependent and
design intended
and return and designed
the
a
any
a a
d ummytransf infologrecordtranslator drawevent enable encodedgroups endiancoder entity epubelementtree.get erfaastrominterpolator errorlist discretize_linear_1d dialog dynamicmatrixtransform indexlocator indexingadapter simpleuploadedfile ipv4 integerlist isortdriver itemsequence itrs jitter join jsonreporter key_field klass illegalseconderror image imageconverter `relative_part` implicitarray import improperlyconfigured imread includenode foreignkeyrawidwidget diamond dictvectorizer did_you_mean diff_treestructure sessionredirectmixin sessionmock sigmaclip representationinfo reportermodify repetition renderdatatree sampproxytimeouterror render relative regularpolycollection redirecturlmixin redirectsession height sample_without_replacement safe_name safe_exists runresult runandparse rugplot set_source_info sip setdiff1d sessionmiddleware differentialattribute direct est discretize_oversample_1d disk2d diverging_palette divider docstringcomponents dodge dotted_netmask doublelowlevelwcs fill ffmpegbase fifty_percent_off figurecanvas simplepostview simplenorm simplelistfilter simplechoiceiterator simpleblogserializer silentdoesnotexist shapedlikendarray seealso sampwebclient sankey.add format_array_flat format_items wcs.fix wcs warning_message warning waiteradmin versionmodified version vartype value validator validationassertions userschema's update_template_context format_row formatter fraction unzip uninstall_repl_displayhook forgivinglatextranslator full_exception_context functionaltestfile hyperlinkavailabilitychecker get_netrc_auth h5netcdfstore grouper gridspecfromsubplotspec gridlinescollection gridhelperrectlinear greaterthan grand_child get_transformer get_node_location haircolorenum histogram httpbasicauth httpresponse huemapping i18ntags rrulelocator function value_field inverted invertedhammertransform invertedmercatorlatitudetransform invocationparams ipinterface session separation_3d selflistschema self.message_equal self._schema self._invoke_load_processors selectinventoryform select_step_degree typealiasnamespace securecookiesessioninterface seasonsgroup scryptpasswordhasher school schemawithname schema invaliduserinput uniformtrirefiner introduction get_lock_path zaxis zarrbase wcs.slice y-coordinate xkcd_palette writerouter wrap windowsconsoleio_workaround idmaker wgs84geodeticrepresentation wcspixel2worldtransform wcsaxes yearend functionaltestreporter gcrs_to_cirs_mat generative get_dataset_names get_default_username get_fixed_timezone get_label ipv4interface roll rendercontext ribbonboximage `noneof` eventcollection `node` `next` `offset` `plot_examples` `physical_lines_for_line` `phase_spectrum` `pathpatcheffect` `path` `patch` `partialarticleformwithslug` `partial` `orderfields` `parentschema.blubb` `parent` `param` `paddedbox` `override` `outer.inners` `nonseq` `noop` `normalize_kernel` `now` subquery subplothost subclassedarray reprexceptioninfo strsequenceorset streamconsumederror stop stepvaluevalidator stdcapturefd stacking `parse_node` stackedbytesarray struct.struct `option` `open_groups` `only` `oneof` `on_rm_rf_error` `on_draw` `on_commit` `on_bind_field` `np_corr_ind` sum_func `mockmovetrackorient` `missing_reference` `multi_jaccard_score` `mtime` `mpl_palette` `moviewriterregistry` `move_legend` `month_anchor_check` `monkeypatch.context` `my_decorator` `robust_getitem` `reversed` `retype` `resolve_reference_detect_inventory` `resolve_color` `resize` `replicate` `render_maths_to_base64` `plot` `remove_transform` `remove_tag` `remoteuserbackend` `releaseevent` `multiplelocaleactivationtestcase` `mod` `my_difficult_property` `myint` `minversion` `min` `messages` `messagedata` `merge_coordinates_without_align` `newcmp` `max` `mark_inset` `map_dataarray_line` `missing` `make_retry_after_handler` exception `make_cube` `mean_squared_log_error` expect_exception `new_line` `nestedinlinetransform` `ndim` `nantag` `name` `myuserform` `make_ds` symlink_or_skip squaredstretch table extractminute thingitem thing thetaformatter textfileform's textarea testschemadeserialization symlognorm testrouter's sip.inverse testmetaclass testclass test_func test_field_deserialization_with_user_validator_that_raises_error_with_list test terminalreporter temppathfactory font fnmatch_ex floatformat float rgbimagemapping fileavoidwrite fileextensionvalidator filefield filepathfield filescompleter fetch_rcv1_fxt fit_transform temporaryfileuploadhandler fitinfoarraycontainer rgb_to_jch rgb_to_huslp resolvedgrouper resolve_time_unit_from_attrs_dtype resampling requiredebugtrue rickerwavelet1d flatblogschema fixedlocator testpluckschema testrunnerbase hexbin skipped sleep slidergroup song sortmore source_branch thumbnail tooltogglebase sqrt spymanager spy teecaptureio truncsecond splitdatetimefield sphinxtranslator sizemapping sphinxrole time trunchour sphinxdoctestrunner trapezoid1d siteregistry transformgraph transformer toprequest top toomanyredirects threadsafeparser toolcontainerbase timestamp timesince timeresampler time_support tick
aaa
use imp lem ` ent atio n
`copy
reuse
inherita
`id` `match`
registration `load` `predict_pro `dontreadfro redefinition `safe_getatt ba` `__init__` minput` `append` formatter r` function shared
exceptions
of
class function function decorator
a function class class and function function responsible and class's class class class class class class class class class class class class tightly class instance class method class function function function function class function method function and method class function's method class function function fixture function function function function function attribute function's function's method function and function function decorator estimator's
class function class library property parameter method function class function function attribute class function and function method function class module function between and mixin
designed function method definition class output class method decorator's function class function function import duplication
object function class definition function function class function method method function
to
attribute class function function function class function's function method method method function function class being method function's attribute function's function function decorator
and function field function class method function class method
of
class function function function method method function function function function function hook function function function parameter attribute method attribute function class attribute entity function function function function's method function's function method function method function function's function function function method class class method
chunks even when definitions and usages span multiple segments, forming a semantic graph that supports higher-level analyses of dependency and modular structure across the codebase.
F gure 4 D s r bu on o he firs our words o a ques ons represen ng he r requency pa erns Emp y co ored b ocks nd ca e suffixes ha are oo rare o show nd v dua y Answer Generat on Answers were genera ed by he same mode used or ques on crea on cond oned on he ques on and re evan code chunks hen refined or conc seness hrough a o ow-up ca Promp va d y was ver fied by human rev ew on a subse be ore sca ng o he u da ase D stractor Creat on and Sty st c A gnment Three d s rac ors per ques on were genera ed n a s ng e ca cond oned on he ques on code and correc answer ensur ng echn ca p aus b y wh e reflec ng common m sconcep ons A er va da ng orma comp ance and card na y correc answers were adap ed o ma ch he ngu s c s y e o d s racors preven ng sur ace- eve pa ern ma ch ng A genera on and adap a on used he same mode w h human-ver fied promp s gu d ng each s age
4.
Experimental Setup
We eva ua e a co ec on o anguage mode s on SWE-QA o assess he r mu -hop code comprehens on capab es and he per nence o our corpus The eva ua ed mode s span rom sma o arge sca es and nc ude wo SmolLM2 var an s Llama3.3-70B-Instruct and DeepSeek-R1 To soa e he effec o reason ng on mu -hop ques on answer ng we nc ude bo h reason ng and nonreason ng var an s o he same mode s such as Phi-4-mini and Qwen3-4B A eva ua ons are conduc ed n a zero-sho se ng mode s rece ve on y he re evan code chunks he ques on and
Oracle Question Answering. In the first setting, models are evaluated with an oracle retrieval. For each question, the code chunks that are relevant are solely provided, without any distractors. Models must return a single answer based on these oracleprovided chunks, the question, and the multiplechoice options. This setting establishes an upper bound on comprehension performance by removing retrieval errors. Retrieval-Based Evaluation. The second setting evaluates retrieval performance. All code chunks of each repository are embedded using the Salesforce/SFR-Embedding-Code400M_R (Liu et al., 2025) model which is optimized for code/text retrieval. For each question, a maximum inner product search is performed on the repository’s vector database to retrieve the ten most relevant chunks. Retrieval quality is measured using NDCG@k and Precision@k, providing insight into the difficulty of locating relevant information compared to the ideal oracle scenario. Noisy Oracle Comprehension. Finally, we test the best-performing models under a noisy oracle setting. Each question is presented with the relevant chunks plus a set of distractor chunks. These distractors are selected based on the most likely but irrelevant chunks retrieved in the second experiment. This setting evaluates whether models can still leverage the correct answer when the context is contaminated with retrieval-induced noise, better reflecting real-world code understanding pipelines. Dataset Validation and Correction LLM-based answer generation occasionally produces incorrect labels. To improve label quality, we employed consensus-based validation on the oracle question answering results. We excluded three small models (SmolLM2-360M, SmolLM2-1.7B, DeepSeek-R1Distill-Qwen-1.5B) to prevent capacity-related noise from corrupting consensus signals. Using two criteria: fewer than 3 models selecting the generated answer, and at least 11 of 12 retained models agreeing on an alternative. This identified 66 candidate mislabeled questions. Spot-check validation on 10
randomly sampled questions confirmed that the consensus answer was correct in all cases (100% precision), justifying batch correction of all 66 questions. All results reported below reflect corrected ground truth labels, ensuring performance metrics reflect genuine code comprehension rather than label artifacts.
5.
Experiments
All results reported below reflect corrected ground truth labels after applying our validation procedure described in Section 4. 1200 1000 Number of Questions
the multiple-choice options—without any additional in-context examples. SWE-QA comprises 9,072 questions drawn, including 4,584 DC questions and 4,488 IE questions. We report overall accuracy, category-specific accuracy, repository-level performance, and retrieval quality metrics such as NDCG@k and Precision@k, where applicable. Post-processing scripts normalize model outputs to extract the selected option, addressing cases where responses include explanations or formatting variations.
800 600 400 200 0
0
1
2
3
4 5 6 7 8 9 10 11 12 13 14 15 Number of Models That Answered Correctly
Figure 5: Histogram showing question difficulty, measured by the number of models that answered each question correctly. Fewer correct responses indicate higher difficulty.
5.1.
Oracle Question Answering
Overall Performance. Across 15 models, accuracy spans 74.41%–19.94%, highlighting the difficulty of multi-hop code reasoning. Llama-3.370B-Instruct leads with 74.41%, 2.4 points above gemma-3-4b-it, demonstrating that optimized smaller models can approach large-scale performance. Qwen3-4B-Instruct follows at 70.05%, while Llama-3.2-3B-Instruct shows rare balance across question types, hinting at architecture-driven inter-entity strengths. Figure 5 shows a broad distribution of question difficulty: nearly half of the questions are correctly answered by at least 10 models, while roughly one-tenth are solved by at most three, providing a meaningful spectrum to differentiate model capabilities. Architecture and Reasoning Effects. The two MoE model families evaluated underperform their dense counterparts across scales. gpt-oss20b variants cluster around 60% accuracy, 10 points below dense models like gemma-3-4b-it, with negligible variation across reasoning depths.
80 70
Accuracy (%)
60 50
Qwen3-4B-Instruct Llama-3.2-3B-Instruct Qwen3-4B-Thinking
20
Llama-3.3-70B-Instruct
gpt-oss-20b high gpt-oss-20b medium gpt-oss-20b low
Phi-4-mini-Reasoning Phi-4-mini-Instruct
DeepSeek-R1
Qwen3-1.7B DeepSeek-R1-Distill-Qwen-1.5B
40 30
gemma-3-4b-it
SmolLM2-1.7B-Instruct
Dense MoE Reasoning Non-Reasoning
SmolLM2-360M-Instruct 100
101
Model Size (Billion Parameters)
102
103
Figure 6: Model accuracy in the Oracle Question Answering setting as a function of parameter count. Circle markers denote dense architectures, squares indicate MoE models. Green corresponds to reasoningtuned models, while blue represents standard instruction-tuned models. DeepSeek-R1, despite 671B total and 37B active parameters, achieves only 60.98%, suggesting a potential scaling plateau not observed in the dense models evaluated. One possible explanation is that multi-hop reasoning demands unified knowledge access, while MoE routing may fragment information across expert boundaries, making crosscontext integration more difficult. Reasoning-enhanced variants show inconsistent benefits. Qwen3-4B-Thinking underperforms its instruct counterpart by 7.6 points, particularly on IE questions, indicating that extended reasoning at small scales may amplify error propagation. Phi-4Mini gains only two points from reasoning, showing limited benefit under capacity constraints. Based on the models evaluated, neither extended reasoning nor larger MoE scale appears to consistently improve multi-hop comprehension, though broader evaluation across more model families would be needed to substantiate this observation. Category-Specific Patterns. As shown per Table 1, models consistently perform better on DC than IE questions, with gaps going from 5 to 15 points. Reasoning models such Qwen3-4BThinking, DeepSeek-R1, and gpt-oss-20b show the largest disparities, confirming that reasoning over interacting entities across three code
Model Llama-3.3-70B-Instruct gemma-3-4b-it Qwen3-4B-Instruct Llama-3.2-3B-Instruct Qwen3-4B-Thinking gpt-oss-20b (medium) DeepSeek-R1 gpt-oss-20b (low) gpt-oss-20b (high) Phi-4-Mini-Reasoning Phi-4-Mini-Instruct Qwen3-1.7B DeepSeek-R1-Dist-Qwen-1.5B SmolLM2-1.7B-Instruct SmolLM2-360M-Instruct
DC (%) 78.75 75.32 74.58 66.79 69.85 67.95 68.23 67.62 67.51 59.77 58.09 48.32 46.72 33.92 19.28
IE (%) 69.98 68.60 65.41 65.50 54.81 53.83 53.58 53.43 53.16 50.35 47.90 43.18 39.14 29.94 20.61
Table 1: Category-level accuracy results for Oracle Question Answering, showing performance on DC and IE questions. locations is substantially harder than single-entity tracing. Top performers such as Llama-3.370B-Instruct, gemma-3-4b-it, and Qwen34B-Instruct exhibit smaller gaps (6–9 points), while Llama-3.2-3B-Instruct stands out with nearly balanced results, suggesting architectural features that enhance inter-entity reasoning even at small scale.
Metric Precision@k Recall@k F1@k Hit Rate@k MRR@k NDCG@k
k=3 0.2288 0.3305 0.2643 0.5909 0.4945 0.3430
k=5 0.1626 0.3875 0.2248 0.6620 0.5108 0.3726
k = 10 0.0961 0.4535 0.1567 0.7341 0.5206 0.3998
Table 2: Retrieval-based evaluation results with k = 3, 5, 10. Model
Overall (%)
DC (%)
IE (%)
Llama-3.3-70B
74.31 ▼ 0.1
77.94 ▼ 0.81
70.61 ▲ 0.63
gemma-3-4b-it
70.63 ▼ 1.37
73.97 ▼ 1.55
67.22 ▼ 1.38
Qwen3-4B-Instruct
68.79 ▼ 1.26
73.01 ▼ 1.57
64.48 ▼ 0.93
Table 3: Difference of model performance in Noisy Oracle Question Answering compared to Oracle Question Answering Model Size vs. Performance Analysis. Performance scales with size but is conditioned by architecture and training methods. Llama-3.3-70BInstruct leads, confirming benefits from scale, though gemma-3-4b-it’s near parity shows the power of optimization. DeepSeek-R1’s weak result despite 37B active parameters shows that scale alone is insufficient. The 3–4B range emerges as a performance “sweet spot,” showing the widest variance driven by architecture and inference strategy. Below 2B parameters, accuracy collapses indicating a lower bound near 3B parameters of current models for effective multi-hop reasoning.
5.2.
Retrieval-Based Evaluation
Table 2 shows moderate retrieval effectiveness. Precision@k declines from 0.23 (k = 3) to 0.10 (k = 10), while Recall@k increases from 0.33 to 1 0.45, reflecting the typical precision-recall tradeoff. The highest F1@k of 0.26 at k = 3 indicates smaller retrieval sets better balance relevance and noise. Hit Rate@k above 0.59 and stable MRR@k around 0.5 show relevant chunks frequently appear in top ranks, though NDCG@k (0.34–0.40) suggests limited ranking quality. Retrieval is sufficient for downstream tasks but leaves room for improvement.
5.3.
Distractor and Oracle Comprehension
Table 3 reveals distinct robustness patterns under retrieval noise. Llama-3.3-70B-Instruct shows negligible degradation (0.04-point drop), with slight DC decline but improved IE performance, suggesting noise can aid distractor elimination.
gemma-3-4b-it degrades modestly (∼1 point) across categories, while Qwen3-4B-Instruct exhibits the largest decline yet retains strong accuracy. Noise tolerance scales with model size, though architecture also matters.
6. 6.1.
Discussion
Architectural Insights
MoE vs. Dense Models. In our experiments, the two MoE model families tested underperform their dense counterparts, potentially pointing to structural limitations in multi-hop reasoning, though caution is warranted given the small sample. DeepSeek-R1 achieves only 60.98%, similar to gpt-oss-20b and below dense counterparts like Llama-3.3-70B-Instruct and gemma-34b-it. This parity across 4–37B active parameters may suggest a scaling plateau not observed in dense architectures, though differences in training data and optimization could equally account for the gap. Possible contributing factors include fragmented expert knowledge hindering cross-context integration, routing difficulties in activating coherent expert sets, or scaling inefficiency. The wider DC–IE gaps of approximately 14-15 pts in MoE models compared to 6–9 pts in dense models are consistent with this pattern, though they do not establish a causal architectural explanation. Reasoning-Enhanced Inference. Reasoning capacity has not yet shown consistent benefits for multi-hop code comprehension. Qwen34B-Thinking underperforms its non-reasoning counterpart by 7.6 points, Phi-4-Mini gains only marginally, and gpt-oss-20b exhibits nearidentical outcomes across reasoning intensities. These results indicate that extended reasoning, regardless of depth, does not reliably improve performance on this task.
6.2.
Qualitative Analysis of Challenges
To better understand what makes certain questions particularly challenging, we conducted a qualitative analysis of failure patterns. We identify three primary axes of difficulty that characterize the reasoning requirements in multi-hop code comprehension. First, multi-entity reasoning refers to questions requiring simultaneous tracking of multiple interacting entities (classes, functions, variables) and their relationships across code segments. Second, multi-hop reasoning denotes questions demanding a sequential chain of logical steps connecting information across dispersed code locations, where each step depends on the previous one. Third, execution modeling encompasses questions necessitating mental simulation of code execution to
Question
Multi-Entity
Multi-Hop
Execution Model
Does the use of set ylim in the set lim and transforms method interact with the tight layout parameter in a way that affects the appearance of the figure, and if so, what is the expected outcome?
set ylim, set lim and transforms, tight layout, figure object
Setting axis limits triggers transform calculations, which affect layout adjustments and final figure appearance
Trace how axis limit changes propagate through the layout engine
How does the order of inheritance for class D, which inherits from both B and C, affect the type checking and instantiation of its subclasses E and F?
Classes B, C, D, E, F, and their inheritance relationships
Method Resolution Order (MRO) of D affects type checking for E and instantiation of F
Execute Python’s MRO algorithm and track type resolution
Does the use of filter metadata in routing methods in the provided code introduce unnecessary dependencies or side effects that could impact the performance or behavior of the ConsumingRegressor.fit method?
Filter function, routing methods, metadata objects, ConsumingRegressor, fit method
Filtering metadata affects routing, which impacts the data available to fit
Trace metadata flow through filtering, routing, and consumption
Table 4: Examples of challenging questions requiring combinations of reasoning capabilities. Questions shown were answered correctly by at most two models, both top performers, demonstrating the difficulty frontier of current multi-hop code comprehension. trace value transformations, state changes, or control flow across multiple components. These axes are not mutually exclusive; the most challenging questions typically combine all three dimensions. To illustrate these reasoning requirements, we analyzed questions that were answered correctly by at most two models, and only by top performers (Llama-3.3-70B-Instruct or gemma-34b-it). This selection criterion eliminates questions likely solved by random guessing while focusing on tasks that remain challenging yet theoretically within reach of current capabilities. Table 4 presents three representative examples exhibiting distinct patterns of complexity. The first question requires understanding how set_ylim method calls propagate through transform calculations to affect layout adjustments—exemplifying multi-hop reasoning through a rendering pipeline with moderate entity complexity. The second question demands analysis of Python’s Method Resolution Order (MRO) across a five-class inheritance hierarchy, requiring precise execution modeling of type resolution alongside tracking multiple class relationships. The third question combines all three dimensions: tracking metadata flow through filtering and routing functions while modeling side effects on the fit method’s behavior. These examples reveal that model failures often stem not from inability to understand individual code constructs, but from difficulty maintaining coherent reasoning chains across multiple logical steps while simultaneously tracking entity states and simulating execution semantics. The architecture-specific performance gaps observed in our quantitative results (Section 6.1) likely reflect differential capabilities along these reasoning axes, with the MoE mod-
els in our sample tending to struggle more when entity tracking and execution modeling must be performed jointly, possibly reflecting routing constraints across expert boundaries.
7.
Conclusion
We introduced SWE-QA, a benchmark for multihop code comprehension that evaluates language models on complex reasoning tasks across real software repositories. SWE-QA captures core challenges of large-scale code understanding and mirrors the cognitive demands faced by developers navigating extensive codebases. The evaluation of fifteen models shows that multihop reasoning remains a major obstacle, with persistent weaknesses in handling dispersed and interdependent code contexts. Our analysis highlights three primary axes of difficulty: multi-entity reasoning, multi-hop reasoning, and execution modeling, revealing gaps in current architectures and training approaches. Future work should expand human annotation to strengthen label reliability, explore systematic prompt design to isolate methodological effects, and extend SWE-QA to additional programming languages. Controlled comparisons across architectures, targeted fine-tuning, and human baselines can further clarify the limits of current models. Explicit modeling of entity relationships and execution dynamics may ultimately enable more structured and execution-aware reasoning. Overall, this benchmark establishes a foundation for advancing study of complex code comprehension and guiding the development of models with deeper compositional reasoning capabilities.
8.
Limitations
Dataset Construction. SWE-QA is restricted to Python repositories from SWE-bench and to 2–3 hop reasoning chains, limiting generalization to other programming languages and to deeper multihop scenarios. Complex code patterns such as asynchronous control flow, metaprogramming, and cross-module dynamic dispatch are largely absent, as they fall outside what static syntactic analysis can reliably model. Questions were generated by Llama-3.2-3B-Instruct; larger generation models may produce higher-quality or more diverse questions. Fixed 1000-character chunking may not optimally suit all code structures or context windows, and training data contamination for the evaluated models cannot be fully excluded. AST-Based Generation Bias. Entity extraction relies on Python’s built-in AST parser, which captures only statically analyzable, syntactically explicit relationships. As a result, benchmark questions are inherently grounded in the structural view of code that the AST provides: declared entities, explicit call sites, and syntactic inheritance chains. This introduces a systematic bias: models and systems that navigate code through similar structural or symbolic lenses may be artificially advantaged. In particular, DC questions, which trace a declaration to its call site, directly mirror the entity relationships that AST-based code navigation tools expose. Systems with access to AST-backed tooling, such as language servers or MCP-enabled tools offering go-todefinition and find-references capabilities, operate on the same representation used to generate the questions and therefore have an inherent structural advantage. Conversely, dynamic code behaviors invisible to static parsing, including runtime polymorphism, decorator-induced modifications, and reflective patterns, are systematically underrepresented. Results should therefore be interpreted with the caveat that the benchmark measures comprehension of statically identifiable code structure, and comparisons involving AST-tool-assisted systems may not reflect genuine differences in code understanding. Evaluation Protocol. All models were evaluated in zero-shot settings without systematic prompt engineering, potentially underestimating models sensitive to prompt format or those that benefit from structured reasoning. Though gpt-oss-20b was tested across thinking intensities, other models used default settings and reasoning hyperparameters were not exhaustively explored. Retrieval used a single embedding model, leaving open whether alternative retrieval strategies would change the relative difficulty of the retrieval-based setting.
Architectural Conclusions. MoE underperformance was observed on only two model families, limiting the strength of architectural generalizations. Correlations between architecture and performance do not establish causality: observed differences may reflect training data composition, optimization choices, or scale rather than fundamental architectural constraints. Claims about MoE limitations should therefore be treated as preliminary observations warranting controlled investigation rather than definitive conclusions.
9.
Ethical Considerations
This work uses publicly available code repositories and focuses on advancing code comprehension capabilities. All code processing and analysis respect the original licenses and usage terms of the source repositories.
10.
Bibliographical References
Jacob Austin et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Mark Chen et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Zhaoling Chen, Xiangru Tang, Gangda Deng, Fang Wu, Jialong Wu, Zhiwei Jiang, Viktor Prasanna, Arman Cohan, and Xingyao Wang. 2025. Locagent: Graph-guided llm agents for code localization. Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024), pages 16568–16621. PMLR. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations. Jade Liu et al. 2021. Codeqa: A question answering dataset for source code comprehension. Findings of EMNLP. Nelson F. Liu, Alex Tamkin, and et al. 2023. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172.
Ye Liu, Rui Meng, Shafiq Joty, Silvio Savarese, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. 2025. Codexembed: A generalist embedding model family for multiligual and multi-task code retrieval.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600.
Shuai Lu et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664.
Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jianguo Li, and Rui Wang. 2023. Unifying the perspectives of nlp and software engineering: A survey on language models for code. arXiv preprint arXiv:2311.07989.
Wei Ma, Shangqing Liu, Zhihao Lin, Wenhan Wang, Qiang Hu, Ye Liu, Cen Zhang, Liming Nie, Li Li, and Yang Liu. 2023. Lms: Understanding code syntax and semantics for code analysis. arXiv preprint arXiv:2305.12138. Xinyu She, Yue Liu, Yanjie Zhao, Yiling He, Li Li, Chakkrit Tantithamthavorn, Zhan Qin, and Haoyu Wang. 2023. Pitfalls in language models for code intelligence: A taxonomy and survey. arXiv preprint arXiv:2310.17903. Changyoon Sohn et al. 2022. Cs1qa: A dataset for assisting code-based question answering in an introductory programming course. NAACL. FAIR CodeGen team, Jade Copet, Quentin Carbonneaux, Gal Cohen, Jonas Gehring, Jacob Kahn, Jannik Kossen, Felix Kreuk, Emily McMilin, Michel Meyer, Yuxiang Wei, David Zhang, Kunhao Zheng, Jordi Armengol-Estapé, Pedram Bashiri, Maximilian Beck, Pierre Chambon, Abhishek Charnalia, Chris Cummins, Juliette Decugis, Zacharias V. Fisches, François Fleuret, Fabian Gloeckle, Alex Gu, Michael Hassid, Daniel Haziza, Badr Youbi Idrissi, Christian Keller, Rahul Kindi, Hugh Leather, Gallil Maimon, Aram Markosyan, Francisco Massa, Pierre-Emmanuel Mazaré, Vegard Mella, Naila Murray, Keyur Muzumdar, Peter O’Hearn, Matteo Pagliardini, Dmitrii Pedchenko, Tal Remez, Volker Seeker, Marco Selvi, Oren Sultan, Sida Wang, Luca Wehrstedt, Ori Yoran, Lingming Zhang, Taco Cohen, Yossi Adi, and Gabriel Synnaeve. 2025. Cwm: An open-weights llm for research on code generation with world models. Harsh Trivedi et al. 2022. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics. Johannes Welbl et al. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics. Tony Xiao, Ankit Patel, Jascha Sohl-Dickstein, and Zhenhai Chen. 2024. Scaling context length in language models: A practical investigation. arXiv preprint arXiv:2402.10590.
A. Benchmark
Comparison to Existing Benchmarks Task
Scope
Multi-hop
Cross-file
Real Repos
CodeQA CS1QA CRUXEval Code World Models CodeXGLUE HumanEval MBPP SWE-bench LocAgent
QA QA I/O Prediction World Modeling Various Generation Generation Patch Gen. Localization
Snippet Snippet Function Function File Function Function Repository Repository
✗ ✗ ✗ ✗ ✗ ✗ ✗ Implicit Implicit
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓
✗ ✗ ✗ ✗ Partial ✗ ✗ ✓ ✓
SWE-QA (Ours)
MCQ
Repository
✓
✓
✓
Table 5: Comparison of existing code benchmarks across key dimensions. SWE-QA is the only benchmark to jointly support explicit multi-hop reasoning, cross-file context, and real repository grounding.
B.
Prompt Specifications
This section documents all prompts used in the QuestionMaker module. For each prompt, we explicitly distinguish between: • System Instructions — Instructions sent as the system message • User Message — Content sent as the user message Placeholders enclosed in {} are programmatically substituted before sending. An explanation of the placeholders is available in Section B.10.
B.1.
Entity-Specific Question Generation
Type: System + User message Purpose: Generate a focused question about a specific entity System Instructions You will be given one or more code snippets, possibly from multiple files. A specific entity (such as a class, function, or variable) will be identified. Entity of Focus: {entity_name} Task: - Write one clear and concise question about this entity. - The question should highlight something a developer might consider, such as purpose, behavior, interactions, or improvements. - Keep the question short and direct. - Do not explain the code or provide an answer. Output format: Question: <your question here>
User Message <JOINED CODE CHUNKS CONTAINING ENTITY>
B.2.
Interacting Entities Question Generation
Type: Single user prompt (no separate system message) Purpose: Generate a question about the interaction between two entities
User Prompt Template User Prompt You are given two code entities, {entity_A} and {entity_B}, along with a snippet where they interact. Your task is to write one clear and concise question about their relationship. Input: - {entity_A} Definition Code: {entity_A_definition_code} - {entity_B} Definition Code: {entity_B_definition_code} - Interaction Code: {entity_interaction_code} Guidelines: - Ask about design, abstraction, dependencies, or side effects. - Keep the question short and direct. - Do not explain the code or provide answers. Output: Question: <your question here>
B.3.
Question Extraction
Type: Single user prompt Purpose: Extract only the final question from generated text User Prompt Template User Prompt Extract only the question from the following text. Return the question exactly, with no extra words or labels: {generated_text}
B.4.
Answer Generation for Code Comprehension
Type: System + User message Purpose: Generate a detailed answer to a comprehension question System Instructions You are an expert in evaluating code comprehension. The user will provide code and a question about it. Your goal is to generate one relevant answer in English. The answer should focus on: - Essential mechanisms of how the code works - Important design decisions - Potential pitfalls or unexpected behaviors
Provide a clear and thorough answer demonstrating deep understanding.
User Message <JOINED CODE CHUNKS> <Question about the code>
B.5.
MCQ Answer Sanitization
Type: Single user prompt Purpose: Convert a verbose answer into a concise MCQ-style answer User Prompt You are an expert Python developer and technical writer. I will give you: 1. A Python code snippet 2. A question about that code 3. A detailed answer Your task is to sanitize the answer: - Remove fluff and redundancy - Keep only what directly answers the question - Make it short, clear, and direct - Do not repeat the question - Do not rephrase the code Input Code: {code} Question: {question} Original Answer: {answer} Sanitized Answer:
B.6.
Distractor Generation for MCQs
Type: Single user prompt Purpose: Generate exactly three plausible distractor answers for a programming MCQ User Prompt You are an expert MCQ generator specializing in programming assessments. Given the following: Code: {code} Question: {question} Correct Answer:
{answer} Generate exactly 3 plausible distractor answers (incorrect but believable options) to be used in a multiple-choice question. Each distractor should: 1. Be contextually relevant to the code and question 2. Represent a different level of Bloom’s Taxonomy (e.g., Understanding, Applying, Analyzing) 3. Be plausible -- choices a well-meaning but mistaken student might select 4. Be similar in structure or terminology to the correct answer 5. Avoid being trivially or obviously incorrect Return ONLY the distractors as a valid Python list of dictionaries: [ { "option": "Distractor text here", "bloom_level": "Bloom’s taxonomy level (e.g., Understanding, Applying, Analyzing)" }, ... ]
B.7.
Correct Answer Adaptation to Match Distractor Style
Type: Single user prompt Purpose: Rephrase the correct answer to match the style and structure of generated distractors User Prompt You are an expert MCQ generator specializing in programming assessments. Given the following: Code: {code} Question: {question} Correct Answer: {answer} Here are 3 distractor answers generated for this question: {distractor_examples} Rephrase the correct answer so that it resembles the distractors in style, structure, and terminology, but remains fully correct. Return ONLY the adapted answer as a string, with no extra explanation or formatting.
B.8.
Multiple-Choice Question Prompt for Model Benchmarking
Type: Single user prompt Purpose: Ask a model to select the correct answer letter (A–D) for a code-related MCQ
User Prompt You are given a piece of code, a related question, and four multiple-choice options labeled A through D. Analyze the code, read the question carefully, and choose the correct answer by responding with only the letter (A, B, C, or D). Code: {code} Question: {question} Options: {formatted_options} Answer (respond with A, B, C, or D only):
Note: The formatted_options placeholder contains the code snippet, question text, and options dictionary formatted as shown in Section B.10.
B.9.
Answer Extraction Using Helper Model
Type: System + User messages Purpose: Extract only the correct answer letter from a model’s response, optionally processing reasoning output System Instructions Extract the letter corresponding to the correct answer from the following response. The output must be only the letter, with no extra explanation or characters.
User Message Sequence User Message Example conversation history: 1. User: "The answer to the question is A." Assistant: "A" 2. User: "B." Assistant: "B" 3. User: "C" Assistant: "C" Followed by the actual response to extract: {conclusion}
Note: The conclusion is extracted from the benchmark model output, with optional removal of <think>...</think> tags if reasoning extraction is enabled.
B.10.
Placeholders Explained
{entity_name} The name of the specific entity being analyzed {entity_A}, {entity_B} Names of two interacting entities
{entity_A_definition_code} Code defining entity A {entity_B_definition_code} Code defining entity B {entity_interaction_code} Code showing how the entities interact {generated_text} Text from which to extract a question {code} The source code snippet for the MCQ {question} The text of the multiple-choice question {answer} The correct answer to the question {distractor_examples} The three generated distractor options {formatted_options} Options A–D formatted as: A. option_text B. option_text C. option_text D. option_text {conclusion} The benchmark model output after optional thought extraction {llm_output} Raw model output potentially containing <think> tags All placeholders are replaced with actual content before sending to the model.