Skip to content

feat: add torch compile#11

Open
JINO-ROHIT wants to merge 2 commits into
mainfrom
torch-compile
Open

feat: add torch compile#11
JINO-ROHIT wants to merge 2 commits into
mainfrom
torch-compile

Conversation

@JINO-ROHIT
Copy link
Copy Markdown
Owner

  1. adding torch compile degrades performance, possible reasons could be loops and modules that cannot be compiled.
  2. try to compile at the leaf node level instead of applying at the top level model

@JINO-ROHIT JINO-ROHIT self-assigned this Mar 25, 2026
@JINO-ROHIT JINO-ROHIT linked an issue Mar 25, 2026 that may be closed by this pull request
@JINO-ROHIT
Copy link
Copy Markdown
Owner Author

  1. we have a lot of graphs being compiled seperately and lots of recompilation, major modules like rope, for loops, kv cache etc is slowing this down a lot.
  2. lots of modules needs to be improved to get maximum performance.
  3. dynamic shapes and sizes also cause recompilation
  4. try using different modes of compilation
  5. do this torwards the end when all techniques have been implemented.

@JINO-ROHIT JINO-ROHIT added the waiting for update waiting for the contributor to update the PR label Mar 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting for update waiting for the contributor to update the PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test effect of torch compile and provide results

1 participant