Skip to content

Add CI and release workflows #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
imenelydiaker wants to merge 9 commits into
base: main
Choose a base branch
from
Draft

Add CI and release workflows #9

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
750a72b
add CI and release workflows
imenelydiaker Jun 23, 2024
1c11f66
Merge branch 'main' into add-ci-realease-workflows
imenelydiaker Jun 24, 2024
6018110
Add linting
imenelydiaker Jun 24, 2024
78143b0
Add cache for dependencies
imenelydiaker Jun 24, 2024
290cb64
upgrade chroma
imenelydiaker Jun 24, 2024
dffa2e6
Fix dependencies and tests
imenelydiaker Jun 24, 2024
b977f2c
Fix fairseq dependency
imenelydiaker Jun 24, 2024
66b98e0
Fix fairseq dependency
imenelydiaker Jun 24, 2024
0b5c2c3
Merge branch 'add-ci-realease-workflows' of https://github.com/Lyon-N…
imenelydiaker Jun 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: CI

on:
push:
branches: [main]
pull_request:

jobs:
tests:
strategy:
matrix:
os: [ubuntu-latest, windows-laest, macos-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- name: Install Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
cache: 'pip'
- name: Install the project dependencies
run: |
pip install --upgrade pip
make install-dev
- name: Run lint
run: |
make lint-check
- name: Run tests
run: |
make tests
34 changes: 34 additions & 0 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: Release

on:
release:
types: [released]

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
cache: 'pip'
- name: Install pypa/build
run: pip install build --user
- name: Install the project dependencies
run: make install
- name: Build wheel and tarball
run: python -m build . --outdir dist/

pypi-publish:
name: Publish to PyPI
needs:
- build
runs-on: ubuntu-latest
permissions:
# IMPORTANT: this permission is mandatory for trusted publishing
id-token: write
steps:
- name: Publish package distributions to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
4 changes: 2 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,5 @@ lint-check:
ruff format . --check
ruff check **/*.py

test:
pytest -n auto
tests:
pytest -n 5
2 changes: 1 addition & 1 deletion chromacache/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
from .chromacache import ChromaCache
from .chromacache import *
from .embedding_functions import *
13 changes: 2 additions & 11 deletions chromacache/chromacache.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,5 @@
try:
import chromadb
from chromadb import Documents, EmbeddingFunction, Embeddings
except Exception:
__import__("pysqlite3")
import sys

sys.modules["sqlite3"] = sys.modules.pop("pysqlite3")

import chromadb
from chromadb import EmbeddingFunction
import chromadb
from chromadb import Documents, EmbeddingFunction, Embeddings


class ChromaCache:
Expand Down
10 changes: 1 addition & 9 deletions chromacache/embedding_functions/AbstractEmbeddingFunction.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,3 @@
try:
__import__("pysqlite3")
import sys

sys.modules["sqlite3"] = sys.modules.pop("pysqlite3")
except Exception:
pass

from abc import ABC, abstractmethod

import tiktoken
Expand Down Expand Up @@ -70,7 +62,7 @@ def __call__(self, input: Documents) -> Embeddings:
"""Wrapper that truncates the documents, encodes them

Args:
documents (Documents): List of documents
input (Documents): List of documents

Returns:
Embeddings: the encoded sentences
Expand Down
10 changes: 10 additions & 0 deletions chromacache/embedding_functions/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,13 @@
from .OpenAIEmbeddingFunction import OpenAIEmbeddingFunction
from .SentenceTransformerEmbeddingFunction import SentenceTransformerEmbeddingFunction
from .VoyageAIEmbeddingFunction import VoyageAIEmbeddingFunction

__all__ = [
"AbstractEmbeddingFunction",
"CohereEmbeddingFunction",
"LaserEmbeddingFunction",
"MistralAIEmbeddingFunction",
"OpenAIEmbeddingFunction",
"SentenceTransformerEmbeddingFunction",
"VoyageAIEmbeddingFunction",
]
19 changes: 11 additions & 8 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
requires = ["setuptools>=69.0", "wheel"]
build-backend = "setuptools.build_meta"

[tool.setuptools.packages.find]
exclude = ["tests"]

[project]
name = "chromacache"
version = "0.0.1"
Expand All @@ -22,12 +25,15 @@ dependencies = [
"voyageai>=0.1.6",
"tiktoken>=0.5.2",
"openai>=1.6.1",
"chromadb>=0.4.21",
"chromadb>=0.5.3",
"python-dotenv>=1.0.1",
"cohere==4.41",
"laser_encoders==0.0.1",
"sentence-transformers==2.2.2",
"mistralai==0.0.12"
"cohere>=4.41",
"laser_encoders>=0.0.1",
"sentence-transformers>=2.2.2",
"mistralai>=0.0.12",
"fairseq>=0.12",
"hydra-core==1.3.2",
"omegaconf==2.3.0"
]

[project.urls]
Expand All @@ -36,9 +42,6 @@ Homepage = "https://github.com/Lyon-NLP/chroma_cache"
[project.optional-dependencies]
dev = ["ruff>=0.0.254", "pytest", "pytest-xdist"]

[tool.setuptools.packages.find]
exclude = ["tests", "results"]

[tool.ruff]
target-version = "py38"

Expand Down
2 changes: 1 addition & 1 deletion tests/embedding_functions/test_AbstractEmbeddingFunction.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ def test_truncate_documents(tokenizer, sentences):
)

assert len(truncated_sentences) == len(sentences)

tokenized_truncated_sentences = [tokenizer.encode(s) for s in truncated_sentences]
for truncated_sentence in tokenized_truncated_sentences:
assert len(truncated_sentence) <= max_token_length