Feature to add a filtering options on the API call

[Nevsksar]

Description

When dealing with enterprise workspaces in ClickUp, as ClickUp admins, we often find ourselves having access to multiple workspaces, lists, tasks, and other elements that are not needed in our pipeline. To save resources, processing time, and improve efficiency, it is beneficial for us to have the ability to filter specific elements that we want to fetch.

Noteworthy ClickUp API Bugs

During the development of this project, I encountered some inherent bugs in ClickUp's API. Primarily, these bugs prevented the passing of a single space or list as a parameter. However, a workaround was discovered: by passing the same ID twice, the API would function correctly, and there are not any duplicate values in the API response.

Issues

1. The following changes have been implemented and work, but they are currently coded to only fetch data from the general settings in the meltano.yml file. Consequently, it is not possible to configure different pipelines for various environments because they are not being correctly read.

For example, when using the command meltano --environment=dev run tap-clickup ..., the configurations are not being applied as expected from the dev envv because it reads only from the common configurations.

Taking this meltano.yml sample, the code always reads "spaces_id" as '90100424497' even when we run it in prod (which in this case should be '90070255202'

version: 1
default_environment: prod
project_id: 8f483ca8-e13c-4abb-8d24-666666666
environments:
- name: prod
  config:
    plugins:
      extractors:
      - name: tap-clickup
        variant: autoidm
        pip_url: tap-clickup
        config:
          workspace_id: 30979640
          spaces_id: '90070255202'
          list_ids: ''
        select:
        - space.name
        - space.id
        - task.*
      loaders:...
plugins:
  extractors:
  - name: tap-clickup
    variant: autoidm
    pip_url: tap-clickup
    config:
      workspace_id: 30979640
      spaces_id: '90100424497'
      list_ids: ''
    select:
    - space.name
    - space.id
    - task.*
  loaders:...

2. Could not figure out how to correctly add the new setting in the meltano.yml file. At first I was just hardcoding it to the file, then I found out about tap-clickup--autoidm.lock file and in this file I added the new configuration settings to be added when using meltano config tap-clickup -set. But I cannot figure out how to edit this into the projects code, i basically added these configs to the og file:

. . .
    {
      "name": "stream_map_config",
      "kind": "object",
      "label": "Stream Map Config",
      "description": "User-defined config values to be used within map expressions."
    },
    {
      "name": "stream_maps",
      "kind": "object",
      "label": "Stream Maps",
      "description": "Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html)."
    },
	    {
      "name": "workspace_id",
      "kind": "integer",
      "label": "Workspace Id",
      "description": "To limit from which workspace to fetch data pass it's id(https://dev-doc.clickup.com/333/d/h/ad-455845/c464293970a6906)."
    },
	    {
      "name": "spaces_id",
      "kind": "string",
      "label": "Spaces Ids",
      "description": "To limit from which Spaces to fetch data from, pass their ids as a list string Example: spaceid1,spaceid2,ispaced3."
    },
	    {
      "name": "list_ids",
      "kind": "string",
      "label": "Lists Ids",
      "description": "To limit from which Lists to fetch data from, pass their ids as a list string Example: listid1,listid2,listid3."
    }
  ]
}

[visch]

This should be an ArrayType see https://sdk.meltano.com/en/latest/classes/typing/singer_sdk.typing.ArrayType.html

[visch]

Params should be handled via https://sdk.meltano.com/en/latest/classes/singer_sdk.RESTStream.html#singer_sdk.RESTStream.get_url_params

[visch]

Did they add a new concept of workteamid? Or workspace_id, could you link to that in clickups docs?

[Nevsksar]

So the structure is Workspace>Space>Folder>List>Task>Subtask

In the app/platform they are called workspaces, but for the API they are called Teams, also there is a feature called "teams" (that group users together) but in the API they call it "group"

https://clickup.com/api/clickupreference/operation/CreateTeam/

[visch]

All of this needs to go away. Please go through the getting started for Meltano, and read through how singer works.

Adding extra fields to a config block added here https://github.com/AutoIDM/tap-clickup/blob/main/meltano.yml#L6 will pass all of the configs via the config CLI argument (via a json file) to the tap. This is all handled by the config options you altered in tap.py below

[visch]

Revert this please

[visch]

Ditto

[visch]

Ditto

[visch]

See below

[visch]

All changes from line 0 to line 50 need to go away