v0.9.0 - #76
Open
earmingol wants to merge 14 commits into
Open
Conversation
…tensor is not alphabetically sorted
The Python 3.12 test job installs pandas 3.0, since pandas 3.0 requires Python >= 3.11 while the 3.10 job stays on pandas 2.3. That surfaced three incompatibilities, for 52 failures and 24 errors on that job alone: - Copy-on-write is mandatory in pandas 3.0, so `DataFrame.values` returns a read-only array. Six places mutated it in place, either zeroing a diagonal or shuffling it, and raised "underlying array is read-only". Copying the dataframe beforehand does not help, because the copy's `.values` is read-only as well. Diagonals are now zeroed through a new `zero_diagonal` helper, and the remaining places copy the array explicitly. This also makes `compute_linkage` stop modifying a dataframe passed by the user, which it did only for dataframes and not for arrays. - `DataFrame.applymap` was removed. The p-value transform of the dot plot is vectorized instead, which works on every supported pandas. - `str` is the default dtype in pandas 3.0, so the values of a single column are an extension array, which has no `.flatten()`. The gene names in `permute_ppi_labels` are now collected with `to_numpy(dtype=object)`. Repairs two latent bugs found in the same code paths. `pcoa(inplace=True)` was dead code, as it went through `np.float`, an alias numpy removed in 1.24, and once repaired it needed a writable array too. `check_presence_in_dataframe` now normalizes a bare column name into a list, as `shuffle_cols_in_df` does. Every change behaves the same on pandas 1, 2 and 3, so no dependency pin changes. Verified against pandas 3.0.5 and 2.3.3 on Python 3.12: 621 passing on both, from 600 before, with the added tests failing on the previous code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`test_regressions_v090.py` was the only file in tests/ not named after a module. It grouped tests by the defect they reproduce rather than by the code under test, so its 37 cases were spread across ten modules and its three pcoa cases sat on concerns that test_external.py already owned. Each case now lives in the file for the module it exercises. Also splits test_external.py per module, as tests/ already does for preprocessing, so pcoa, umap, gseapy and goenrich each have their own file. pcoa_utils goes with pcoa, the way test_tensor_helpers.py groups the small tensor modules. Test bodies, names, parametrizations and markers are carried over unchanged; only the call style is adapted to the imports of each destination. The comment blocks explaining each defect move with their tests, since they document why the assertions exist. One case is dropped rather than moved: test_reorder_dimension_elements_without_metadata was a strict subset of test_reorder_dimension_elements in test_plotting.py, which makes the same call and additionally checks that the other dimensions keep their order. Its explanation of the bug is folded into the test that stays. Verified against pandas 3.0.5 and 2.3.3 on Python 3.12: 620 passing on both, from 621, the difference being the dropped duplicate. The other 36 cases were each confirmed to still be collected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`matplotlib.cm.get_cmap` was deprecated in matplotlib 3.7 and removed in 3.11, breaking the three dot plot tests. As with pandas, the Python 3.12 job is the only one affected, because matplotlib 3.11 requires Python >= 3.11 and the 3.10 job is held at 3.10.x, where the function still exists. Uses the pyplot function instead, which `aesthetics.py` and `circular_plot.py` already use, is not deprecated on any version down to the declared floor of matplotlib 3.2, and accepts a Colormap as well as a name, as the removed one did. `matplotlib.cm.ScalarMappable`, used further down the same function, is unaffected. Verified on Python 3.12 against matplotlib 3.11.1 with pandas 3.0.5, and against matplotlib 3.10.8 with pandas 3.0.5 and 2.3.3: 620 passing on all three. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three calls warned that they will not survive seaborn 0.14. Unlike the pandas and matplotlib breakages, these would have taken down both CI jobs at once, as seaborn resolves to the same version on either Python. - `sns.distplot` in `pvalue_from_dist` becomes `sns.histplot(kde=True, stat='density')`. distplot normalized the histogram to a density whenever it drew a KDE, ignoring `norm_hist=False`, so `stat='density'` is what reproduces it. Checked on seaborn 0.13.2, which still has both: same bin count, same bar heights, same area of 1.0, same single KDE line and same colour, so the `axvline` drawn from `get_lines()[-1]` is unchanged too. - The boxplot of `context_boxplot` passed a palette with no `hue`. It now repeats the x variable as the hue and turns the legend off, which is what seaborn recommends and what keeps the plot identical: the boxes are not dodged because the two variables are redundant, and the x axis already labels the groups. - `umap_biplot` passed its `cmap` as a palette even when `hue` was None, where seaborn ignored it with a warning. The palette is now only passed with a hue. The boxplot change needs the `legend` option of the categorical plots, added in seaborn 0.13.0, so the floor in setup.py moves from 0.11.0 to 0.13.0. Verified that 0.12.2 has no such option, and that its `dodge` default of True would have split every box in two, so there is no form of this that works on both. Verified on Python 3.10 and 3.12 with the dependencies each job resolves: 620 passing on both, and none of the three warnings left. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`pip install -e .[test]` was unconstrained, so the versions CI tested against changed whenever anything in the dependency tree released. That turned the build red twice in two days with no commit to this repository: pandas 3.0, then matplotlib 3.11, both reaching only the Python 3.12 job because both require Python >= 3.11 while the 3.10 job is held a generation behind. Adds a constraints file per matrix leg, holding the versions each job resolves today, and installs with `-c`. Builds become reproducible and dependencies move when the files are regenerated, which the header of each explains how to do. The pins apply to CI only; setup.py stays open, so nothing here restricts what a user installing cell2cell gets. Pinning alone would mean never hearing about a new release, so a second job runs the same suite with nothing pinned. It is scheduled weekly and can be triggered by hand, and deliberately does not run on pushes or pull requests, because a failure there is a dependency needing attention rather than a broken commit. Verified that both constraints files resolve from a clean interpreter and pin exactly what they list, with nothing floating outside them: 65 packages on 3.12 and 55 on 3.10, 620 tests passing in each. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit fixed the palette-without-hue call in `context_boxplot` with `hue=x, legend=False`, as seaborn recommends, and raised the floor in setup.py to 0.13.0 because that is when `legend` was added to the categorical plots. That narrowed what users can install for no good reason: seaborn 0.13 needs Python >= 3.8, which would have made it the strictest requirement in the package and dropped support for Python 3.7. Before, the strictest were `pandas >= 1.0.0` and `gseapy >= 1.0.3`, both of which reach 3.7. `dodge=False` achieves the same thing and has existed since long before 0.13, so the floor goes back to 0.11.0 and no Python version is lost. The legend that the hue brings is dropped afterwards instead of by an argument. Verified on seaborn 0.12.2 as well as 0.13.2 that the boxes land at the same centres, with the same widths and colours, and no legend, exactly as when the palette was passed alone. `legend=False` raises TypeError on 0.12.2, which is what ruled it out. setup.py is now identical to v0.9.0 again. Verified on Python 3.10 and 3.12, with both the pinned and the newest dependencies: 620 passing in all four. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release Notes - cell2cell v0.9.0
New features
New
'trimean'aggregation method for single cells:aggregate_single_cellsincell2cell.preprocessing.rnaseqacceptsmethod='trimean'to aggregate the gene expression of thesingle cells composing a cell type using Tukey's trimean (
0.5 * Q2 + 0.25 * (Q1 + Q3)), computedwith a new
cell2cell.preprocessing.rnaseq._trimeanhelper. It is a weighted average of the medianand the first and third quartiles, so it is more robust to outliers than the average while still
accounting for the spread of the distribution, and it ignores NaNs. It is also available through the
aggregation_methodparameter ofSingleCellInteractionsincell2cell.analysis.New dependency
natsort: Element names are now sorted using natural sorting (natsorted)instead of lexicographic sorting (
sorted) across the package. Names containing numbers are orderedas expected (e.g.
Factor 1, Factor 2, ..., Factor 10instead ofFactor 1, Factor 10, Factor 2, ...,and
C-1, C-2, C-10instead ofC-1, C-10, C-2).Feature updates
Natural sorting of element names:
sorted()was replaced withnatsorted()in the functionsthat define the order of element names, including
build_context_ccc_tensoranddataframes_to_tensorin
cell2cell.tensor(tensor gene, cell, LR-pair and context orders),add_sliding_window_info_to_adatain
cell2cell.spatial(window column names),run_label_permutationincell2cell.stats,initialize_interaction_spaceincell2cell.analysis,get_filtered_ppi_networkandget_genes_from_go_hierarchyincell2cell.preprocessing,get_files_from_directoryincell2cell.io,aggregate_single_cellsincell2cell.preprocessing(order of the aggregated cell types),and
generate_legend,pcoa_3dplot,context_boxplotandcircos_plotincell2cell.plotting.Reproducible order of elements: Multiple functions obtained the order of their elements from a
Python
set, which is not reproducible across runs. These now return a deterministic order:generate_pairsincell2cell.core.interaction_spacekeeps the order given by the list of cellswhen removing duplicated pairs, which also defines the column order of the communication matrix.
get_element_abundancesincell2cell.preprocessing.find_elementskeeps the order in whichelements are first found, making the
how='outer'options of the tensor builders reproducible.dataframes_to_tensorincell2cell.tensorkeeps the order in which elements appear in theinput dataframes when
sort_elements=False.random_switching_ppi_labelsincell2cell.stats.permutationsorts the genes before permutingthem, so results are now reproducible for a given
random_statewhen a list ofgenesis passed.circos_plotandcontext_boxplotincell2cell.plottingproduce a reproducible order ofcells, nodes and groups.
Note on the order of results: Because of the changes above, the order of the elements in some
outputs may differ from previous versions (tensor dimensions, columns of the communication matrix,
order of groups and legends in plots, and permutations obtained with a fixed
random_state). Thevalues themselves are unchanged. Lexicographic sorting was intentionally kept in
remove_ppi_bidirectionalityincell2cell.preprocessing.ppibecause it determines which directionof a bidirectional interaction is kept, so its results remain identical to previous versions.
Fixed Bugs
Wrong cell-pair labels in
flatten_factor_ccc_networks: This function incell2cell.analysis.tensor_downstreambuilt the names of the sender-receiver pairs from thesorted names of the cells, but flattened the loadings of the factor-specific networks in the order
the cells have in the tensor. When the elements of the sender and receiver dimensions of the tensor
were not alphabetically sorted, every loading was assigned to the wrong sender-receiver pair. The
names of the cell pairs are now built directly from the adjacency matrices, keeping the order of the
tensor dimensions, and the matrices are reindexed before flattening so that values and labels always
match.
get_lr_by_cell_pairsused this function internally, so its results were affected too.Results and figures generated from tensors whose sender/receiver elements were not alphabetically
sorted should be regenerated. Results obtained from tensors with alphabetically sorted elements
were not affected and are identical to previous versions.
Factor order in tensor downstream analyses:
get_factor_specific_ccc_networksandcompute_gini_coefficientsincell2cell.analysis.tensor_downstreamsorted the factor nameslexicographically, so decompositions with 10 or more factors were returned in the order
Factor 1, Factor 10, Factor 11, Factor 2, .... They are now naturally sorted.Factor names in
get_lr_by_cell_pairs: This function assumed that every factor was named'<word> <integer>'and raised an error with any other naming. It now uses natural sorting insteadof parsing the factor names.
aggregate_single_cellsmodified the dataframe passed by the user: This function incell2cell.preprocessing.rnaseqreplaced the index of the input dataframe with the cell types andadded a
'celltype'column to it, modifying it in place. As a consequence, calling the functiontwice on the same dataframe raised a
KeyError. It now groups the single cells by an external listof cell types, leaving the input untouched (without copying the expression matrix). The aggregated
values are unchanged; only the column order may differ, as the cell types are now naturally sorted.
add_sliding_window_info_to_adatawith pandas >= 2.0: This function incell2cell.spatialpassed a
setof barcodes toDataFrame.loc, which newer versions of pandas do not accept. Thebarcodes are now passed as a list.
Docstring of
flatten_factor_ccc_networks: It stated that rows were factors and columns werecell-cell pairs, while the returned dataframe contains the cell-cell pairs as rows and the factors
as columns.