Skip to content

Week 3 #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sleepypioneer wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Week 3 #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d892638
add level 1 and 2 for project 2
sleepypioneer May 7, 2023
e20a3bc
update results
sleepypioneer May 10, 2023
8f79e7c
Merge remote-tracking branch 'origin/main' into week_3
sleepypioneer May 10, 2023
e6b1a77
add week 3
sleepypioneer May 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,3 +127,6 @@ dmypy.json

# Pyre type checker
.pyre/

# data
datasets
1,008 changes: 1,008 additions & 0 deletions poetry.lock
View file
Open in desktop

Large diffs are not rendered by default.

24 changes: 24 additions & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
[tool.poetry]
name = "search-engineering-main"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"
packages = [{include = "search_engineering_main"}]

[tool.poetry.dependencies]
python = "3.9.7"
opensearch-py = "^2.2.0"
requests = "^2.29.0"
ipython = "^8.13.1"
urljoin = "^1.0.0"
pandas = "^2.0.1"
fastparquet = "^2023.4.0"
click = "^8.1.3"
lxml = "^4.9.2"
kaggle = "^1.5.13"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
93 changes: 93 additions & 0 deletions week2/results.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Week 2 Results

## Level 1

### Q: How long did it take to index the 1.2M product data set?  What docs/sec indexing rate did you see?

4.30 minutes
5.74K docs/sec

### Q: Notice that the Index size rose (roughly doubled) while the content was being indexed, peaked, then ~ 5 minutes after indexing stopped, the index size dropped down substantially.  Why did it drop back down?  (What did OpenSearch do here?)

The on-disk size of the index roughly doubled because we were indexing the exact same content. When indexing content, even if it's previously indexed content, OpenSearch is creating new Lucene segments on disk to store those documents, this is due to the append-only structure of Lucene segments meaning documents are not updated in-place. The index size dropped down again because OpenSearch merges these segments as part of the process and then removes the older copies of documents.

Takeaway: This is a good example of how disk usage can burst substantially when you’re re-indexing (i.e. updating) a large percentage of the docs in your index.  This is why it’s important to have enough disk space to handle these bursts.

### Q: Looking at the metrics dashboard, what queries/sec rate are you getting?

137 queries/sec

### Q: What resource(s) appear to be the constraining factor?

The OpenSearch Process CPU is at 100%. We also see a maximum of 2 Threads in the thread pool for search. This is because the search [thread pool](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/modules-threadpool.html) will only increase to two threads on a system with 1 CPU. 

## Level 2

#### CPUs 2, Memory 2GB

Some spikes up to 4 threads for search, however in parts it only used 3. The Queue depth was also 0 throughout. I noticed that the CPU usage relative to the request was doubled from 20% to 40%.

I created a new panel for this using the following promql query:

`sum(irate(container_cpu_user_seconds_total{}[5m])) by (namespace,container) / sum(container_spec_cpu_shares{} / 1024) by (namespace,container)`

and memory is almost at 100% relative to the limit.

`sum(container_memory_usage_bytes{}) by (namespace,container) / sum(container_spec_memory_limit_bytes{}) by (namespace,container)`

I noticed CPU usage is peaking far beyond the limit set (a max of 300%!!) I used the following query for this:

`sum(irate(container_cpu_usage_seconds_total{}[5m])) by (namespace,container) / sum(container_spec_cpu_quota{} / container_spec_cpu_period{}) by (namespace,container)`

Also noticeable is the search query rate jumped to 289 queries/sec.


#### Index query rate in relation to resources


| CPU count | Memory | CPU peak relative to request | CPU mean relative to request | CPU peak relative to limit | Memory peak relative to limit | Index query rate max | Thread pool | Queue depth | GC |
| --------- | ------ | ----------------------------- | ----------------------------- | --------------------------- | ------------------------------ | --------------------- | ----------- | ----------- | -- |
| 1 | 2GB | 20% | 17% | 216% | 98% | 5.11k | 1 | 14 | 3 |
| 2 | 2GB | 38% | 28% | 214% | 108% | 7.65k | 2 | 10 | 1 |
| 2 | 4GB | 40% | 34% | 210% | 90% | 8.63k | 3 | 12 | 1 |
| 2 | 8GB | 40% | 27% | 218% | 105% | 7.14k | 2 | 13 | 3 |
| 4 | 2GB | 72% | 60% | 189% | 109% | 8.60k | 5 | 2 | 1 |
| 4 | 4GB | 38% | 37% | 206% | 92% | 8.44k | 4 | 8 | 1 |
| 4 | 8GB | 70% | 40% | 196% | 103% | 8.27k | 5 | 11 | 1 |

#### Search query rate in relation to resources

| CPU count | Memory | CPU peak relative to request | CPU mean relative to request | CPU peak relative to limit | Memory peak relative to limit | Search query rate max | Thread pool | Queue depth | GC |
| --------- | ------ | ----------------------------- | ----------------------------- | --------------------------- | ------------------------------ | --------------------- | ----------- | ----------- | -- |
| 1 | 2GB | 20% | 18% | 102% | 97% | 137 | 2 | 2 | 2 |
| 2 | 2GB | 40% | 32% | 209% | 94% | 297 | 4 | 0 | 3 |
| 2 | 4GB | 71% | 55% | 197% | 81% | 420 | 4 | 1 | 1 |
| 2 | 8GB | 40% | 30% | 212% | 78% | 290 | 4 | 0 | 2 |
| 4 | 2GB | 70% | 48% | 185% | 103% | 462 | 4 | 2 | 1 |
| 4 | 4GB | 68% | 47% | 182% | 90% | 377 | 4 | 1 | 0 |
| 4 | 8GB | 39% | 38% | 209% | 82% | 224 | 4 | 1 | 1 |

### Q: As you increased CPU and memory in your L2 tests, what seemed to be the constraining factor limiting indexing rate?

It appears that the increase of both CPUs and memory had a positive impact on the indexing rate.

### Q: As you increased CPU and memory in your L2 tests, what was the constraining factor for querying rate?

As the CPU increased it was noticeably how the query rate increased. Memory seemed to make little difference.

## Level 3

### Q: What is the impact on your query throughput (QPS) and indexing throughput (docs/sec)?

## Level 4 (Optional)

Can you break the system? Can you move the needle on a given metric? Given what you’ve learned about the various system metrics, the way Lucene and OpenSearch works as well as your general knowledge of CPU, Memory and I/O, go through the metrics in our class dashboard and see if you can purposefully move the needle on that metric via indexing or querying changes and not via resource constraints on the container.


Ideas to try:
- Aggressively try different aggregations to induce field data changes
- Try span, regex and other really expensive queries
- Write your own indexer and query client that generates way more load
- Call your search engine from other machines to generate more load?
- Try restricting some of the lower level settings related to threads, memory, etc.
- Try a different garbage collector
8 changes: 5 additions & 3 deletions week3/bbuy_products.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
{
"settings": {
"index": {
"number_of_shards": 3,
"number_of_replicas": 0
},
"analysis": {
"analyzer": {
"smarter_hyphens": {
Expand Down Expand Up @@ -249,7 +253,6 @@
"suggest": {
"type": "completion"
}

}
},
"onSale": {
Expand Down Expand Up @@ -282,7 +285,6 @@
"suggest": {
"type": "completion"
}

}
},
"productId": {
Expand Down Expand Up @@ -423,4 +425,4 @@
}
}
}
}
}
205 changes: 205 additions & 0 deletions week3/results.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
# Week 3 Results

## Level 1

### Q1: Which node was elected as cluster manager?

**Answer:**

Opensearch-node1 was elected as cluster manager.

`GET _cat/nodes?v`

| ip |heap.percent |ram.percent |cpu | load_1m | load_5m | load_15m | node.role | node.roles | cluster_manager | name |
|---------------------|-------------|------------|----|---------|---------|----------|-----------|-----------------------------------------------------|------------------|-----------------------|
| 172.18.0.7 | 20 |98 | 2 | 0.35 | 0.27 | 0.28 | dimr | cluster_manager,data,ingest,remote_cluster_client | * | opensearch-node1 |
| 172.18.0.6 | 45 |98 | 3 | 0.35 | 0.27 | 0.28 | dimr | cluster_manager,data,ingest,remote_cluster_client | - | opensearch-node2 |
| 172.18.0.5 | 52 |98 | 3 | 0.35 | 0.27 | 0.28 | dimr | cluster_manager,data,ingest,remote_cluster_client | - | opensearch-node3 |


### Q2: After stopping the previous cluster manager, which node was elected the new cluster manager?

**Answer:**

Opensearch-node3 was elected as cluster manager.

```sh
docker stop opensearch-node1
```

| ip |heap.percent |ram.percent |cpu | load_1m | load_5m | load_15m | node.role | node.roles | cluster_manager | name |
|---------------------|-------------|------------|----|---------|---------|----------|-----------|-----------------------------------------------------|------------------|-----------------------|
| 172.18.0.5 | 29 |91 | 2 | 0.07 | 0.16 | 0.23 | dimr | cluster_manager,data,ingest,remote_cluster_client | * | opensearch-node3 |
| 172.18.0.6 | 29 |91 | 2 | 0.07 | 0.16 | 0.23 | dimr | cluster_manager,data,ingest,remote_cluster_client | - | opensearch-node2 |

[INFO ][o.o.c.c.FollowersChecker ] [opensearch-node3] FollowerChecker{discoveryNode={opensearch-node1}, failureCountSinceLastSuccess=1, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty
[INFO ][o.o.c.c.Coordinator ] [opensearch-node3] cluster-manager node [{opensearch-node1}] failed, restarting discovery
[INFO ][o.o.c.s.ClusterApplierService] [opensearch-node3] cluster-manager node changed {previous [{opensearch-node1}], current []}, term: 2, version: 66, reason: becoming candidate: onLeaderFailure
[INFO ][o.o.c.s.MasterService ] [opensearch-node3] elected-as-cluster-manager ([2] nodes joined)[{opensearch-node3} elect leader, {opensearch-node2} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 4, version: 67, delta: cluster-manager node changed {previous [], current [{opensearch-node3}]}


### Q3: Did the cluster manager node change again? (was a different node elected as cluster manager when you started the node back up?)

**Answer:**

Opensearch-node3 stays the cluster manager.

```sh
docker start opensearch-node1
```

| ip |heap.percent |ram.percent |cpu | load_1m | load_5m | load_15m | node.role | node.roles | cluster_manager | name |
|---------------------|-------------|------------|----|---------|---------|----------|-----------|-----------------------------------------------------|------------------|-----------------------|
| 172.18.0.7 | 21 |98 | 47 | 0.50 | 0.23 | 0.20 | dimr | cluster_manager,data,ingest,remote_cluster_client | - | opensearch-node1 |
| 172.18.0.5 | 40 |98 | 48 | 0.50 | 0.23 | 0.20 | dimr | cluster_manager,data,ingest,remote_cluster_client | * | opensearch-node3 |
| 172.18.0.6 | 36 |98 | 47 | 0.50 | 0.23 | 0.20 | dimr | cluster_manager,data,ingest,remote_cluster_client | - | opensearch-node2 |


## Level 2: Creating a sharded Index


```
PUT /sensor
{
"settings" : {
"index" : {
"number_of_shards" : 3,
"number_of_replicas" : 2
}
}
}
```

```
curl -k -X PUT -u admin:admin https://localhost:9200/bbuy_products -H 'Content-Type: application/json' -d @week3/bbuy_products.json
cd week3
export BBUY_DATA=../datasets/product_data/products
python index.py -s $BBUY_DATA -w 8 -b 500
```

INFO:Indexing ../datasets/product_data/products to bbuy_products with 8 workers, refresh_interval of -1 to host localhost with a maximum number of docs sent per file per worker of 200000 and 500 per batch.
INFO:Done. 1275077 were indexed in 5.702952890283333 minutes. Total accumulated time spent in `bulk` indexing: 18.04553139530001 minutes

`GET /_cat/shards/bbuy_products?v&s=shard,prirep`

| index | shard | prirep | state | docs | store | ip | node |
|---------------|-------|--------|---------|--------|---------|------------|------------------|
| bbuy_products | 0 | p | STARTED | 423938 | 417.6mb | 172.18.0.7 | opensearch-node1 |
| bbuy_products | 0 | r | STARTED | 423938 | 418.6mb | 172.18.0.6 | opensearch-node2 |
| bbuy_products | 0 | r | STARTED | 423938 | 418.9mb | 172.18.0.5 | opensearch-node3 |
| bbuy_products | 1 | p | STARTED | 425833 | 420.8mb | 172.18.0.6 | opensearch-node2 |
| bbuy_products | 1 | r | STARTED | 425833 | 417.4mb | 172.18.0.5 | opensearch-node3 |
| bbuy_products | 1 | r | STARTED | 425833 | 419.8mb | 172.18.0.7 | opensearch-node1 |
| bbuy_products | 2 | p | STARTED | 425306 | 424.9mb | 172.18.0.5 | opensearch-node3 |
| bbuy_products | 2 | r | STARTED | 425306 | 431.7mb | 172.18.0.6 | opensearch-node2 |
| bbuy_products | 2 | r | STARTED | 425306 | 433.9mb | 172.18.0.7 | opensearch-node1 |


```
{
"settings": {
"index": {
"number_of_shards": 3,
"number_of_replicas": 2
},
...
}
}
```

### Q4: How much faster was it to index the dataset with 0 replicas versus the previous time with 2 replica shards?

INFO:Done. 1275077 were indexed in 2.8425195375000003 minutes. Total accumulated time spent in `bulk` indexing: 4.234756157766661 minutes

**Answer:**

5.70 - 2.84 = 2.86 minutes faster

### Q5: Why was it faster?

Because there is no replica shards, so the data is only written to the primary shard.

Update the existing index to add 2 replica shards.

```
PUT /bbuy_products/_settings
{
"index": {
"number_of_replicas": 2
}
}
```


### Q6: How long did it take to create the new replica shards?  This will be the difference in time between those two log messages.


[2023-05-12T22:59:52,188][INFO ][o.o.c.m.MetadataUpdateSettingsService] [opensearch-node3] updating number_of_replicas to [2] for indices [bbuy_products]
[2023-05-12T23:00:35,129][INFO ][o.o.c.r.a.AllocationService] [opensearch-node3] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[bbuy_products][0]]]).

2023-05-12T23:00:35,129 - 2023-05-12T22:59:52,188 = 43.941 seconds

### Q7: Those two messages were both logged by the cluster_manager.  Why do you think the cluster manager is the node that logs these actions (versus non-manager nodes)?

The cluster manager handles the cluster state, so it is responsible for managing the number of replicas shards and where they are located.

`GET /_cat/shards/bbuy_products?v&s=shard,prirep`

| index | shard | prirep | state | docs | store | ip | node |
|---------------|-------|--------|---------|--------|--------------|------------|------------------|
| bbuy_products | 0 | p | STARTED | 423938 | **440.8mbb** | 172.18.0.7 | opensearch-node1 |
| bbuy_products | 0 | r | STARTED | 423938 | **440.8mb** | 172.18.0.6 | opensearch-node2 |
| bbuy_products | 0 | r | STARTED | 423938 | **440.8mb** | 172.18.0.5 | opensearch-node3 |
| bbuy_products | 1 | p | STARTED | 425833 | **413.5mbb** | 172.18.0.6 | opensearch-node2 |
| bbuy_products | 1 | r | STARTED | 425833 | **413.5mbb** | 172.18.0.5 | opensearch-node3 |
| bbuy_products | 1 | r | STARTED | 425833 | **413.5mb** | 172.18.0.7 | opensearch-node1 |
| bbuy_products | 2 | p | STARTED | 425306 | **412.4mbb** | 172.18.0.5 | opensearch-node3 |
| bbuy_products | 2 | r | STARTED | 425306 | **412.4mbb** | 172.18.0.6 | opensearch-node2 |
| bbuy_products | 2 | r | STARTED | 425306 | **412.4mbb** | 172.18.0.7 | opensearch-node1 |


> *Note that the replicas are now the same storage size as their primaries, since they are now exact copies of (not re-indexed versions of) the primary shard content.*

## Level 3: Query Performance with Replica Shards

``sh
export BBUY_QUERIES=/Users/jessica-g/Documents/search/corise/search_engineering/search_engineering/datasets
python ./query.py -q $BBUY_QUERIES/train.csv -w 4 -m 25000
```

Q: Looking at the metrics dashboard, what queries/sec rate are you getting?

**Answer:**

360 queries/sec


Q: How does that compare to the max queries/sec rate you saw in week 2?

**Answer:**

Max queries/sec rate in week 2: 462 queries/sec (4CPU with 2GB)

My week 2 max is around 100 more queries/sec than week. This doesn't seem to match the suggested 2x improvement in the project instructions.


## Level 4: Re-sharding

> The shrink index API operation moves all of your data in an existing index into a new index with fewer primary shards.

```
PUT bbuy_products/_settings
{
"index.blocks.write": true
}
```

```
POST /bbuy_products/_shrink/bbuy_products
{
"settings": {
"index.number_of_replicas": 2,
"index.number_of_shards": 1
}
}
```