|
55683
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
Edit
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Redo
Share & export
Copy
Show more options
Enter a prompt for Gemini
encrypted
Enter a prompt for Gemini
encrypted
Open upload file menu
Tools
Open mode picker
Pro
Microphone
Send message
Your Jiminny chats aren’t used to improve our models. Gemini is AI. It can make mistakes, so double check it.
Your privacy & Gemini Opens in a new window
Your privacy & Gemini
Opens in a new window
Summarize page
Summarize page
Skip to content
Skip to content
Open menu
Homepage (g then d)
jiminny
jiminny
app
app
Search or jump to…
Type
/
to search
Chat with Copilot
Open Copilot…
Create new...
All issues(g then i)
All pull requests
All repositories
You have unread notifications(g then n)
Open user navigation menu
Repository navigation
Repository navigation
Code
Code
Pull requests (31)
Pull requests
(
31
)
Agents
Agents
Actions
Actions
Wiki
Wiki
Security and quality (1)
Security and quality
(
1
)
Insights
Insights
Settings
Settings
Important update
Important update
On April 24 we'll start using GitHub Copilot interaction data for AI model training unless you opt out.
Review this update
Review this update
and manage your preferences in your
GitHub account settings
GitHub account settings
.
Dismiss banner
JY-20808 low priority indexing queue #12082 Edit title
JY-20808 low priority indexing queue
#...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55683
|
|
55684
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
Edit
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Redo
Share & export
Copy
Show more options
Enter a prompt for Gemini
encrypted
Enter a prompt for Gemini
encrypted
Open upload file menu
Tools
Open mode picker
Pro
Microphone
Send message
Your Jiminny chats aren’t used to improve our models. Gemini is AI. It can make mistakes, so double check it.
Your privacy & Gemini Opens in a new window
Your privacy & Gemini
Opens in a new window
Summarize page
Summarize page
Skip to content
Skip to content
Open menu
Homepage (g then d)
jiminny
jiminny
app
app
Search or jump to…
Type
/
to search
Chat with Copilot
Open Copilot…
Create new...
All issues(g then i)
All pull requests
All repositories
You have unread notifications(g then n)
Open user navigation menu
Repository navigation
Repository navigation
Code
Code
Pull requests (31)
Pull requests
(
31
)
Agents
Agents
Actions
Actions
Wiki
Wiki
Security and quality (1)
Security and quality
(
1
)
Insights
Insights
Settings
Settings
Important update
Important update
On April 24 we'll start using GitHub Copilot interaction data for AI model training unless you opt out.
Review this update
Review this update
and manage your preferences in your
GitHub account settings
GitHub account settings
.
Dismiss banner
JY-20808 low priority indexing queue #12082 Edit title
JY-20808 low priority indexing queue
#
12082
Edit title
Preview
Preview
Awaiting approval
Awaiting approval
Code
Code
Open
Vasil-Jiminny
Vasil-Jiminny
wants to merge 36 commits into
master
master
from
JY-20808-low-priority-indexing-queue
JY-20808-low-priority-indexing-queue
Copy head branch name to clipboard
Lines changed: 432 additions & 159 deletions
Conversation (5)
Conversation
(
5
)
Commits (36)
Commits
(
36
)
Checks (3)
Checks
(
3
)
Files changed (18)
Files changed
(
18
)
Pull Request Toolbar
Pull Request Toolbar
Collapse file tree
Open
JY-20808 low priority indexing queue
JY-20808 low priority indexing queue
#
12082
All commits
All commits
Vasil-Jiminny
Vasil-Jiminny
wants to merge 36 commits into
master
master
from
JY-20808-low-priority-indexing-queue
JY-20808-low-priority-indexing-queue
Copy head branch name to clipboard
0 of 18 files viewed
Awaiting approval
Awaiting approval
Submit review
Submit
review
Open diff view settings
Open overview panel
Open comments panel...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55684
|
|
55688
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;
use Jiminny\Component\ES\QueuePriorityEnum;
/**
/**
* A very basic and simple collection of entity IDs.
* A very basic and simple collection of entity IDs.
*/
*/
class SelectionList
class SelectionList
{
{
public function __construct(
private const int DEFAULT_CHUNK = 200;
private readonly array $priorityEntityIdList = [],
private readonly array $normalEntityIdList = [],
private int $chunkSize = self::DEFAULT_CHUNK;
private readonly bool $isStaleChunkSelected = false,
private bool $isStaleChunkSelected = false;
private readonly int $chunkSize = 200,
private array $selectedEntityIds = [];
) {
// Do nothing
public function setChunkSize(int $chunkSize): void
{...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55688
|
|
55689
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;
use Jiminny\Component\ES\QueuePriorityEnum;
/**
/**
* A very basic and simple collection of entity IDs.
* A very basic and simple collection of entity IDs.
*/
*/
class SelectionList
class SelectionList
{
{
public function __construct(
private const int DEFAULT_CHUNK = 200;
private readonly array $priorityEntityIdList = [],
private readonly array $normalEntityIdList = [],
private int $chunkSize = self::DEFAULT_CHUNK;
private readonly bool $isStaleChunkSelected = false,
private bool $isStaleChunkSelected = false;
private readonly int $chunkSize = 200,
private array $selectedEntityIds = [];
) {
// Do nothing
public function setChunkSize(int $chunkSize): void
{
$this->chunkSize = $chunkSize;
}
}
public function getAllIds(): array
public function markStale(): void
{
{
return array_merge($this->priorityEntityIdList, $this->normalEntityIdList);
$this->isStaleChunkSelected = true;
}
}
public function count(): int
public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void
{
{
return count($this->priorityEntityIdList) + count($this->normalEntityIdList);
$priorityKey = $this->normalisePriority($priority);
if (! array_key_exists($priorityKey, $this->selectedEntityIds)) {
$this->selectedEntityIds[$priorityKey] = [];
}
$this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds);
}
}
public function isEmpty(): bool
/**
* @return string[]
*/
public function getEntityListGroups(): array
{
{
return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList);
return array_keys($this->selectedEntityIds);
}
}
public function getHighPriorityEntityIds(): array
public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array
{
{
return $this->priorityEntityIdList;
$priorityKey = $this->normalisePriority($priority);
return $this->selectedEntityIds[$priorityKey] ?? [];...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55689
|
|
55690
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55690
|
|
55691
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55691
|
|
55694
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;
use Jiminny\Component\ES\QueuePriorityEnum;
/**
/**
* A very basic and simple collection of entity IDs.
* A very basic and simple collection of entity IDs.
*/
*/
class SelectionList
class SelectionList
{
{
public function __construct(
private const int DEFAULT_CHUNK = 200;
private readonly array $priorityEntityIdList = [],
private readonly array $normalEntityIdList = [],
private int $chunkSize = self::DEFAULT_CHUNK;
private readonly bool $isStaleChunkSelected = false,
private bool $isStaleChunkSelected = false;
private readonly int $chunkSize = 200,
private array $selectedEntityIds = [];
) {
// Do nothing
public function setChunkSize(int $chunkSize): void
{
$this->chunkSize = $chunkSize;
}
}...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55694
|
|
55696
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55696
|
|
55697
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55697
|
|
55699
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;
use Jiminny\Component\ES\QueuePriorityEnum;
/**
/**
* A very basic and simple collection of entity IDs.
* A very basic and simple collection of entity IDs.
*/...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55699
|
|
55700
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;
use Jiminny\Component\ES\QueuePriorityEnum;
/**
/**
* A very basic and simple collection of entity IDs.
* A very basic and simple collection of entity IDs.
*/
*/
class SelectionList
class SelectionList
{
{
public function __construct(
private const int DEFAULT_CHUNK = 200;
private readonly array $priorityEntityIdList = [],
private readonly array $normalEntityIdList = [],
private int $chunkSize = self::DEFAULT_CHUNK;
private readonly bool $isStaleChunkSelected = false,
private bool $isStaleChunkSelected = false;
private readonly int $chunkSize = 200,
private array $selectedEntityIds = [];
) {
// Do nothing
public function setChunkSize(int $chunkSize): void
{
$this->chunkSize = $chunkSize;
}
}
public function getAllIds(): array
public function markStale(): void
{
{
return array_merge($this->priorityEntityIdList, $this->normalEntityIdList);
$this->isStaleChunkSelected = true;
}
}
public function count(): int
public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void
{
{
return count($this->priorityEntityIdList) + count($this->normalEntityIdList);
$priorityKey = $this->normalisePriority($priority);
if (! array_key_exists($priorityKey, $this->selectedEntityIds)) {
$this->selectedEntityIds[$priorityKey] = [];
}
$this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds);
}
}
public function isEmpty(): bool
/**
* @return string[]
*/
public function getEntityListGroups(): array
{
{
return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList);
return array_keys($this->selectedEntityIds);
}
}
public function getHighPriorityEntityIds(): array
public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array
{
{
return $this->priorityEntityIdList;
$priorityKey = $this->normalisePriority($priority);
return $this->selectedEntityIds[$priorityKey] ?? [];
}
}
public function getNormalPriorityEntityId(): array
public function getAllIds(): array
{
{
return $this->normalEntityIdList;
$allIds = [];
foreach ($this->selectedEntityIds as $entityIds) {
$allIds = array_merge($allIds, $entityIds);
}
return $allIds;
}
public function count(): int
{
return array_sum(
array_map('count', $this->selectedEntityIds)
);
}
public function isEmpty(): bool
{
return $this->count() === 0;
}
}
public function isStaleChunkSelected(): bool
public function isStaleChunkSelected(): bool
{
{
return $this->count() >= $this->chunkSize;
return $this->count() >= $this->chunkSize;
}
}
private function normalisePriority(string|QueuePriorityEnum $priority): string
{
return $priority instanceof QueuePriorityEnum ? $priority->value : $priority;
}
}
}
app/Component/ES/Processor/Traits/SelectEntityListTrait.php
+27-4Lines changed: 27 additions & 4 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Log;
use Jiminny\Component\ES\Processor\UpdateTarget;
use Jiminny\Component\ES\Processor\UpdateTarget;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidArgumentException;
use Jiminny\Exceptions\InvalidArgumentException;
/**
/**
* Build the correct source list name for the type of entity and priority
* Build the correct source list name for the type of entity and priority
*/
*/
trait SelectEntityListTrait
trait SelectEntityListTrait
{
{
/**
* Highest priority queue for data requiring near-instant availability....
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55700
|
|
55701
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;
use Jiminny\Component\ES\QueuePriorityEnum;
/**
/**
* A very basic and simple collection of entity IDs.
* A very basic and simple collection of entity IDs.
*/
*/
class SelectionList
class SelectionList
{
{
public function __construct(
private const int DEFAULT_CHUNK = 200;
private readonly array $priorityEntityIdList = [],
private readonly array $normalEntityIdList = [],
private int $chunkSize = self::DEFAULT_CHUNK;
private readonly bool $isStaleChunkSelected = false,
private bool $isStaleChunkSelected = false;
private readonly int $chunkSize = 200,
private array $selectedEntityIds = [];
) {
// Do nothing
public function setChunkSize(int $chunkSize): void
{
$this->chunkSize = $chunkSize;
}
}
public function getAllIds(): array
public function markStale(): void
{
{
return array_merge($this->priorityEntityIdList, $this->normalEntityIdList);
$this->isStaleChunkSelected = true;
}
}
public function count(): int
public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void
{
{
return count($this->priorityEntityIdList) + count($this->normalEntityIdList);
$priorityKey = $this->normalisePriority($priority);
if (! array_key_exists($priorityKey, $this->selectedEntityIds)) {
$this->selectedEntityIds[$priorityKey] = [];
}
$this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds);
}
}
public function isEmpty(): bool
/**
* @return string[]
*/
public function getEntityListGroups(): array
{
{
return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList);
return array_keys($this->selectedEntityIds);
}
}
public function getHighPriorityEntityIds(): array
public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array
{
{
return $this->priorityEntityIdList;
$priorityKey = $this->normalisePriority($priority);
return $this->selectedEntityIds[$priorityKey] ?? [];
}
}
public function getNormalPriorityEntityId(): array
public function getAllIds(): array
{
{
return $this->normalEntityIdList;
$allIds = [];
foreach ($this->selectedEntityIds as $entityIds) {
$allIds = array_merge($allIds, $entityIds);
}
return $allIds;
}
public function count(): int
{
return array_sum(
array_map('count', $this->selectedEntityIds)
);
}
public function isEmpty(): bool
{
return $this->count() === 0;
}
}
public function isStaleChunkSelected(): bool
public function isStaleChunkSelected(): bool
{
{
return $this->count() >= $this->chunkSize;
return $this->count() >= $this->chunkSize;
}
}
private function normalisePriority(string|QueuePriorityEnum $priority): string
{
return $priority instanceof QueuePriorityEnum ? $priority->value : $priority;
}
}
}
app/Component/ES/Processor/Traits/SelectEntityListTrait.php
+27-4Lines changed: 27 additions & 4 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Log;
use Jiminny\Component\ES\Processor\UpdateTarget;
use Jiminny\Component\ES\Processor\UpdateTarget;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidArgumentException;
use Jiminny\Exceptions\InvalidArgumentException;
/**
/**
* Build the correct source list name for the type of entity and priority
* Build the correct source list name for the type of entity and priority
*/
*/
trait SelectEntityListTrait
trait SelectEntityListTrait
{
{
/**
* Highest priority queue for data requiring near-instant availability.
* Very few entities should be added here, only when absolutely necessary.
*/
private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority';
/**
* Normal priority queue for normal application operations.
* Almost all update events should pass through this queue
*/
private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update';
private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update';
private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority';
protected static function selectTargetList(string $entityType, bool $withPriority = false): string
/**
{
* Nonblocking low priority queue. This queue will be consumed only when there is nothing...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55701
|
|
55727
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;
use Jiminny\Component\ES\QueuePriorityEnum;
/**
/**
* A very basic and simple collection of entity IDs.
* A very basic and simple collection of entity IDs.
*/
*/
class SelectionList
class SelectionList
{
{
public function __construct(
private const int DEFAULT_CHUNK = 200;
private readonly array $priorityEntityIdList = [],
private readonly array $normalEntityIdList = [],
private int $chunkSize = self::DEFAULT_CHUNK;
private readonly bool $isStaleChunkSelected = false,
private bool $isStaleChunkSelected = false;
private readonly int $chunkSize = 200,
private array $selectedEntityIds = [];
) {
// Do nothing
public function setChunkSize(int $chunkSize): void
{
$this->chunkSize = $chunkSize;
}
}
public function getAllIds(): array
public function markStale(): void
{
{
return array_merge($this->priorityEntityIdList, $this->normalEntityIdList);
$this->isStaleChunkSelected = true;
}
}
public function count(): int
public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void
{
{
return count($this->priorityEntityIdList) + count($this->normalEntityIdList);
$priorityKey = $this->normalisePriority($priority);
if (! array_key_exists($priorityKey, $this->selectedEntityIds)) {
$this->selectedEntityIds[$priorityKey] = [];
}...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55727
|
|
55728
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;
use Jiminny\Component\ES\QueuePriorityEnum;
/**
/**
* A very basic and simple collection of entity IDs.
* A very basic and simple collection of entity IDs.
*/
*/
class SelectionList
class SelectionList
{
{
public function __construct(
private const int DEFAULT_CHUNK = 200;
private readonly array $priorityEntityIdList = [],
private readonly array $normalEntityIdList = [],
private int $chunkSize = self::DEFAULT_CHUNK;
private readonly bool $isStaleChunkSelected = false,
private bool $isStaleChunkSelected = false;
private readonly int $chunkSize = 200,
private array $selectedEntityIds = [];
) {
// Do nothing
public function setChunkSize(int $chunkSize): void
{
$this->chunkSize = $chunkSize;
}
}
public function getAllIds(): array
public function markStale(): void...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55728
|
|
55729
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55729
|
|
55730
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55730
|
|
55731
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;
use Jiminny\Component\ES\QueuePriorityEnum;
/**
/**
* A very basic and simple collection of entity IDs.
* A very basic and simple collection of entity IDs.
*/
*/
class SelectionList
class SelectionList
{
{
public function __construct(
private const int DEFAULT_CHUNK = 200;
private readonly array $priorityEntityIdList = [],
private readonly array $normalEntityIdList = [],
private int $chunkSize = self::DEFAULT_CHUNK;
private readonly bool $isStaleChunkSelected = false,
private bool $isStaleChunkSelected = false;
private readonly int $chunkSize = 200,
private array $selectedEntityIds = [];
) {
// Do nothing
public function setChunkSize(int $chunkSize): void
{
$this->chunkSize = $chunkSize;
}
}
public function getAllIds(): array
public function markStale(): void
{
{
return array_merge($this->priorityEntityIdList, $this->normalEntityIdList);
$this->isStaleChunkSelected = true;
}
}
public function count(): int
public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void
{
{
return count($this->priorityEntityIdList) + count($this->normalEntityIdList);
$priorityKey = $this->normalisePriority($priority);
if (! array_key_exists($priorityKey, $this->selectedEntityIds)) {
$this->selectedEntityIds[$priorityKey] = [];
}
$this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds);
}
}
public function isEmpty(): bool
/**
* @return string[]
*/
public function getEntityListGroups(): array...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55731
|
|
55735
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55735
|
|
55736
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55736
|
|
55737
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;
use Jiminny\Component\ES\QueuePriorityEnum;
/**
/**
* A very basic and simple collection of entity IDs.
* A very basic and simple collection of entity IDs.
*/
*/
class SelectionList
class SelectionList
{
{
public function __construct(
private const int DEFAULT_CHUNK = 200;
private readonly array $priorityEntityIdList = [],
private readonly array $normalEntityIdList = [],
private int $chunkSize = self::DEFAULT_CHUNK;
private readonly bool $isStaleChunkSelected = false,
private bool $isStaleChunkSelected = false;
private readonly int $chunkSize = 200,
private array $selectedEntityIds = [];
) {
// Do nothing
public function setChunkSize(int $chunkSize): void
{
$this->chunkSize = $chunkSize;
}
}
public function getAllIds(): array
public function markStale(): void
{
{
return array_merge($this->priorityEntityIdList, $this->normalEntityIdList);...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55737
|
|
55738
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;
use Jiminny\Component\ES\QueuePriorityEnum;
/**
/**
* A very basic and simple collection of entity IDs.
* A very basic and simple collection of entity IDs.
*/
*/
class SelectionList
class SelectionList
{
{
public function __construct(
private const int DEFAULT_CHUNK = 200;
private readonly array $priorityEntityIdList = [],
private readonly array $normalEntityIdList = [],
private int $chunkSize = self::DEFAULT_CHUNK;
private readonly bool $isStaleChunkSelected = false,
private bool $isStaleChunkSelected = false;
private readonly int $chunkSize = 200,
private array $selectedEntityIds = [];...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55738
|
|
55739
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;
use Jiminny\Component\ES\QueuePriorityEnum;
/**
/**
* A very basic and simple collection of entity IDs.
* A very basic and simple collection of entity IDs.
*/
*/
class SelectionList
class SelectionList
{
{
public function __construct(
private const int DEFAULT_CHUNK = 200;
private readonly array $priorityEntityIdList = [],
private readonly array $normalEntityIdList = [],
private int $chunkSize = self::DEFAULT_CHUNK;
private readonly bool $isStaleChunkSelected = false,
private bool $isStaleChunkSelected = false;
private readonly int $chunkSize = 200,
private array $selectedEntityIds = [];
) {
// Do nothing
public function setChunkSize(int $chunkSize): void
{
$this->chunkSize = $chunkSize;
}
}
public function getAllIds(): array
public function markStale(): void
{...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55739
|
|
55743
|
rirero.• 0Platform Sprint 4 Q2 - Platform TeJY-207 rirero.• 0Platform Sprint 4 Q2 - Platform TeJY-20725 add HS rate limit handlJY-20808 low priority indexing— New Tab• Goodle geminiGeminiPHP• Bottleneck: count() is frequently called (e.g., inside (isEmpty() , isStaleChunkSelected().and potentially loops). Running array_map and array_sum every single time createsunnecessary micro-overhead.• Optimization: Maintain a private int $totalCount = 0; property. Increment it insideaddEntityList() : $this-›totalCount += count(SentityIds); . Then your count()method simply becomes return $this->totalCount;.2. Redis Network BottleneckIn TargetEntitiesSelector: : select() , you are doing sequential round-trips to Redis for eachqueue:1. getStaleChunk() (Network trip)extractidsQueuePrioritvEnum: •HIGH...) Network trio)3. extractIds(QueuePriorityEnum: : NORMAL...› (Network trip)• Optimization (Future): While this PR maintains existing behavior, sequential Redis calls blockexecution. If queue processing speed becomes an issue, consider migrating these extractioncommands to a Redis Lua script or using Pipeline/Transactions to fetch multiple queues in asingle network round-trip.Are you confident that all external calls to methods likeAsvncUpdateElasticSearch::markEntitvForUodate across the entire codebase have been uodatedto pass the new Enum instead of a boolean, or should we add a temporary fallback to support bothtypes during the transition?© Enter a prompt for GeminiProvYour Jiminny chats aren't used to improve our models. Gemini is Al. It can make mistakes, so double check it. Your privacy & GeminiSummarize page87 OpenJY-20808 low orioritv indexing queue #12082|Vasil-Jiminny wants to merge 36 commits into master from JY-20808-low-priority-indexing-queue EQ Filter files...~ E app~ 5 Component/ESv ListenersIE UpdateMultipleTargetsListen...E UpdateSingleTargetListener...• { Processor~ & Actions# UpsertDocumentsAction.phpv • DTOsE SelectionList.php• ^ TraitsSelectEntityListTrait.php# TargetEntitiesSelector.phpAsyncUpdateElasticSearch.php# QueuePriorityEnum.ohnUpdateProcessManager.phpv Console/Commands/Elasticsear…E AsyncUpdateEsEntities.phpE ResetAsyncElasticSearchCom.> Ea tests/Unit/Component/ES|~ app/Component/ES/Processor/TargetEntitiesSelector.php +private function getStaleChunk(): array@ -103,4 +100,12 @@ private function getStaleChunk(): arrayreturn ();106}app/Component/ES/AsyncUpdateElasticSearch.php [C -17,17 +17,20 @@ class AsyncUpdateElasticSearch* Various methods and actions call this method to rebuild auto-score, plays,shares, topic triggers* and other activity related datalpublic static function markEntityForUpdate(string SentityType, int Sentityld, boolSwithPriority = false): void22 -StargetList = self::selectTargetList(SentityType, SwithPriority);Redis::sadd(StaraetList. Sentitvid):Loa::info('| AsvncUodateElasticSearch " Entitv added to Redis list'. [I'entitylype"=> SentityType,30-'entitvid' => Sentitvid.'collectionkey' = $targetList,'withPriority' => $withPriority,< S0 f Daily - Platform-now A) 100% C/ &• Tue 19 May 9:47:38• / 18 viewed • Awaiting approvallSubmit review+26 -21 00200• Viewedprivate function getStaleChunk(): arrayreturn ?:i private tunction extractos(QueuePriorstyEnun Sprdoraty, int SchunkSize): arrayreturn Sthis->extractIdsFromSet(lAsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget,spriority),108 +109110 +SchunkSize+17 -10 0087гViewed1...* Various methods and actions call this method to rebuild auto-score, plays,shares, topic uriggers* and other activity related data20 + Public static function narkentstyForupdatestring $entityType,int $entityld,QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL): void &StaraetList = self::selectTargetList(SentitvTvoe. Soriority):Redis::sadd(StaraetList. SentitvId):33 +Loa:•info('[ AsvncUodateElasticSearch 1 Entity added to Redis list'. ['entityType' = SentityType,'entitvid' => Sentitvid.l'collectionkey' => StargetList,'oriority' => Soriority->value.1):@@ -39,13 +42,13 @ public static function markEntityForUpdate(string $entityType, int $entityId, bopublic static function bulkMarkEntitiesForUpdate(strina SentitvTvoe.array sencitylas,public static function bulkMarkEntitiesForUpdate(string SentityType,array SentityIds,...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55743
|
|
55744
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;
use Jiminny\Component\ES\QueuePriorityEnum;
/**
/**
* A very basic and simple collection of entity IDs.
* A very basic and simple collection of entity IDs.
*/
*/
class SelectionList
class SelectionList
{
{
public function __construct(
private const int DEFAULT_CHUNK = 200;
private readonly array $priorityEntityIdList = [],
private readonly array $normalEntityIdList = [],
private int $chunkSize = self::DEFAULT_CHUNK;
private readonly bool $isStaleChunkSelected = false,
private bool $isStaleChunkSelected = false;
private readonly int $chunkSize = 200,
private array $selectedEntityIds = [];
) {
// Do nothing
public function setChunkSize(int $chunkSize): void
{
$this->chunkSize = $chunkSize;
}
}
public function getAllIds(): array
public function markStale(): void
{
{
return array_merge($this->priorityEntityIdList, $this->normalEntityIdList);
$this->isStaleChunkSelected = true;
}
}
public function count(): int
public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void
{
{
return count($this->priorityEntityIdList) + count($this->normalEntityIdList);
$priorityKey = $this->normalisePriority($priority);
if (! array_key_exists($priorityKey, $this->selectedEntityIds)) {
$this->selectedEntityIds[$priorityKey] = [];
}
$this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds);
}
}
public function isEmpty(): bool
/**
* @return string[]
*/
public function getEntityListGroups(): array
{
{
return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList);
return array_keys($this->selectedEntityIds);
}
}
public function getHighPriorityEntityIds(): array
public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array
{
{
return $this->priorityEntityIdList;
$priorityKey = $this->normalisePriority($priority);
return $this->selectedEntityIds[$priorityKey] ?? [];
}
}
public function getNormalPriorityEntityId(): array
public function getAllIds(): array
{
{
return $this->normalEntityIdList;
$allIds = [];...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55744
|
|
55745
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;
use Jiminny\Component\ES\QueuePriorityEnum;
/**
/**
* A very basic and simple collection of entity IDs.
* A very basic and simple collection of entity IDs.
*/
*/
class SelectionList
class SelectionList
{
{
public function __construct(
private const int DEFAULT_CHUNK = 200;
private readonly array $priorityEntityIdList = [],
private readonly array $normalEntityIdList = [],
private int $chunkSize = self::DEFAULT_CHUNK;
private readonly bool $isStaleChunkSelected = false,
private bool $isStaleChunkSelected = false;
private readonly int $chunkSize = 200,
private array $selectedEntityIds = [];
) {
// Do nothing
public function setChunkSize(int $chunkSize): void
{
$this->chunkSize = $chunkSize;
}
}
public function getAllIds(): array
public function markStale(): void
{
{
return array_merge($this->priorityEntityIdList, $this->normalEntityIdList);
$this->isStaleChunkSelected = true;
}
}
public function count(): int
public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void
{
{
return count($this->priorityEntityIdList) + count($this->normalEntityIdList);
$priorityKey = $this->normalisePriority($priority);
if (! array_key_exists($priorityKey, $this->selectedEntityIds)) {
$this->selectedEntityIds[$priorityKey] = [];...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55745
|
|
55746
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously....
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55746
|
|
55747
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55747
|
|
55748
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;
use Jiminny\Component\ES\QueuePriorityEnum;
/**
/**
* A very basic and simple collection of entity IDs.
* A very basic and simple collection of entity IDs.
*/
*/
class SelectionList
class SelectionList
{
{
public function __construct(
private const int DEFAULT_CHUNK = 200;
private readonly array $priorityEntityIdList = [],
private readonly array $normalEntityIdList = [],
private int $chunkSize = self::DEFAULT_CHUNK;
private readonly bool $isStaleChunkSelected = false,
private bool $isStaleChunkSelected = false;
private readonly int $chunkSize = 200,
private array $selectedEntityIds = [];
) {
// Do nothing
public function setChunkSize(int $chunkSize): void
{
$this->chunkSize = $chunkSize;
}
}
public function getAllIds(): array
public function markStale(): void
{
{
return array_merge($this->priorityEntityIdList, $this->normalEntityIdList);
$this->isStaleChunkSelected = true;
}
}
public function count(): int
public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void
{
{
return count($this->priorityEntityIdList) + count($this->normalEntityIdList);
$priorityKey = $this->normalisePriority($priority);...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55748
|
|
55749
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55749
|
|
55750
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously....
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55750
|
|
55751
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-49a github.com/jiminny/app/pull/12082/changes#diff-49a79d99c97caf7fc27ecb1dbacab5a29d871822caa0db0c71e94957c28c87af...
|
55751
|
|
55692
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Show the uploaded image in a lightbox
Copy prompt
You said Make a PR of changes Find potential issues and bottlenecks
You said
Make a PR of changes Find potential issues and bottlenecks
Listen
Show thinking
Gemini said
Gemini said
I can see you've uploaded a screenshot of a code diff, likely from a version control system given the red and green highlights. I would love to help you review the changes, spot bottlenecks, and help you draft a Pull Request.
However,
the image resolution is far too low for me to read.
It appears to be a completely zoomed-out view where the code has been reduced to tiny, illegible pixels. Because of this, I cannot extract any text or logic to analyze.
How You Can Share the Code
How You Can Share the Code
To help me review your code effectively, please try one of the following:
Paste the text directly:
Copy the specific code blocks (both the old and new versions if possible) and paste them directly into our chat.
Upload the files:
If the changes are extensive, you can upload the raw text or code files.
Take high-resolution screenshots:
If you must use images, take closer, readable screenshots of the specific functions or methods you want me to review.
Once you provide the readable code, I can dive right in to identify potential bugs, assess performance bottlenecks, and help you summarize the changes for your PR!
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said app/Component/ES/Listeners/UpdateMultipleTargetsListener.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners; namespace Jiminny\Component\ES\Listeners; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Jiminny\Contracts\ES\Events\UpdateMultipleEntities; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntities(), $event->getEntities(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } } } app/Component/ES/Listeners/UpdateSingleTargetListener.php +9-1Lines changed: 9 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\Actions\TrackElasticSearchOperations; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Jiminny\Contracts\ES\Events\UpdateSingleEntity; use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface; 'entityId' => $event->getEntityId(), 'entityId' => $event->getEntityId(), ]); ]); /** * @important: * Synchronous event should eventually be deprecated. * Instant write operations are blocking and should be used sparingly. * It is much better in terms of performance to write 100 or 1000 documents at once, * and have one blocking reindex operation, instead of writing 10 documents synchronously. */ if ($event->isSyncEvent()) { if ($event->isSyncEvent()) { $this->handleSyncEvent($event); $this->handleSyncEvent($event); AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( $event->getUpdateTargetValue(), $event->getUpdateTargetValue(), $event->getEntityId(), $event->getEntityId(), $event->isPriority() $event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL, ); ); } } app/Component/ES/Processor/Actions/UpsertDocumentsAction.php +2-1Lines changed: 2 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\DTOs\SimpleCollection; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\ResponseEvaluator; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidDataException; use Jiminny\Exceptions\InvalidDataException; use Sentry\Laravel\Facade as Sentry; use Sentry\Laravel\Facade as Sentry; AsyncUpdateElasticSearch::markEntityForUpdate( AsyncUpdateElasticSearch::markEntityForUpdate( entityType: $this->updateTarget, entityType: $this->updateTarget, entityId: (int) $document->getId(), entityId: (int) $document->getId(), withPriority: true priority: QueuePriorityEnum::HIGH, ); ); } } app/Component/ES/Processor/DTOs/SelectionList.php +54-17Lines changed: 54 additions & 17 deletions Original file line numberOriginal file lineDiff line numberDiff line change namespace Jiminny\Component\ES\Processor\DTOs; namespace Jiminny\Component\ES\Processor\DTOs; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * A very basic and simple collection of entity IDs. * A very basic and simple collection of entity IDs. */ */ class SelectionList class SelectionList { { public function __construct( private const int DEFAULT_CHUNK = 200; private readonly array $priorityEntityIdList = [], private readonly array $normalEntityIdList = [], private int $chunkSize = self::DEFAULT_CHUNK; private readonly bool $isStaleChunkSelected = false, private bool $isStaleChunkSelected = false; private readonly int $chunkSize = 200, private array $selectedEntityIds = []; ) { // Do nothing public function setChunkSize(int $chunkSize): void { $this->chunkSize = $chunkSize; } } public function getAllIds(): array public function markStale(): void { { return array_merge($this->priorityEntityIdList, $this->normalEntityIdList); $this->isStaleChunkSelected = true; } } public function count(): int public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void { { return count($this->priorityEntityIdList) + count($this->normalEntityIdList); $priorityKey = $this->normalisePriority($priority); if (! array_key_exists($priorityKey, $this->selectedEntityIds)) { $this->selectedEntityIds[$priorityKey] = []; } $this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds); } } public function isEmpty(): bool /** * @return string[] */ public function getEntityListGroups(): array { { return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList); return array_keys($this->selectedEntityIds); } } public function getHighPriorityEntityIds(): array public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array { { return $this->priorityEntityIdList; $priorityKey = $this->normalisePriority($priority); return $this->selectedEntityIds[$priorityKey] ?? []; } } public function getNormalPriorityEntityId(): array public function getAllIds(): array { { return $this->normalEntityIdList; $allIds = []; foreach ($this->selectedEntityIds as $entityIds) { $allIds = array_merge($allIds, $entityIds); } return $allIds; } public function count(): int { return array_sum( array_map('count', $this->selectedEntityIds) ); } public function isEmpty(): bool { return $this->count() === 0; } } public function isStaleChunkSelected(): bool public function isStaleChunkSelected(): bool { { return $this->count() >= $this->chunkSize; return $this->count() >= $this->chunkSize; } } private function normalisePriority(string|QueuePriorityEnum $priority): string { return $priority instanceof QueuePriorityEnum ? $priority->value : $priority; } } } app/Component/ES/Processor/Traits/SelectEntityListTrait.php +27-4Lines changed: 27 additions & 4 deletions Original file line numberOriginal file lineDiff line numberDiff line change use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; /** /** * Build the correct source list name for the type of entity and priority * Build the correct source list name for the type of entity and priority */ */ trait SelectEntityListTrait trait SelectEntityListTrait { { /** * Highest priority queue for data requiring near-instant availability. * Very few entities should be added here, only when absolutely necessary. */ private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; /** * Normal priority queue for normal application operations. * Almost all update events should pass through this queue */ private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_NORMAL_PRIORITIES_LIST = '%s-for-update'; private const string ENTITIES_HIGH_PRIORITIES_LIST = '%s-for-update-priority'; protected static function selectTargetList(string $entityType, bool $withPriority = false): string /** { * Nonblocking low priority queue. This queue will be consumed only when there is nothing * with higher priority. * * This queue will be used for operations such as es:reset-async */ private const string ENTITIES_LOW_PRIORITIES_LIST = '%s-for-update-low'; protected static function selectTargetList( string $entityType, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { if (! in_array($entityType, UpdateTarget::allowedTargets())) { if (! in_array($entityType, UpdateTarget::allowedTargets())) { throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); throw new InvalidArgumentException('AsyncUpdateElasticSearch, invalid entity type'); } } if ($withPriority) { if ($priority->isHigh()) { Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); Log::debug('[AsyncUpdateElasticSearch] - Priority check passed and priority is on'); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_HIGH_PRIORITIES_LIST, $entityType); } } if ($priority->isLow()) { return sprintf(self::ENTITIES_LOW_PRIORITIES_LIST, $entityType); } return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); return sprintf(self::ENTITIES_NORMAL_PRIORITIES_LIST, $entityType); } } } } app/Component/ES/Processor/TargetEntitiesSelector.php +26-21Lines changed: 26 additions & 21 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\DTOs\SelectionList; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\ExtractIdsTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait; use Jiminny\Component\ES\QueuePriorityEnum; /** /** * This class supplies a list of entities pending an update. * This class supplies a list of entities pending an update. public function select(): SelectionList public function select(): SelectionList { { $staleWorkEntityIds = $this->getStaleChunk(); $selectionList = new SelectionList(); $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $staleWorkEntityIds = $this->getStaleChunk(); if (count($staleWorkEntityIds) > 0) { if (count($staleWorkEntityIds) > 0) { return new SelectionList( /** priorityEntityIdList: $staleWorkEntityIds, * Stale chunks are treated as high priority isStaleChunkSelected: true, */ chunkSize: ChunkSize::getStaleChunkSize($this->updateTarget), $selectionList->markStale(); ); $selectionList->addEntityList(QueuePriorityEnum::HIGH, $staleWorkEntityIds); $selectionList->setChunkSize(ChunkSize::getStaleChunkSize($this->updateTarget)); return $selectionList; } } $mainChunkSize = ChunkSize::getMainChunkSize($this->updateTarget); $selectionList->setChunkSize($mainChunkSize); $priorityEntityIdsList = $this->extractIdsFromSet( $priorityEntityIdsList = $this->extractIds(QueuePriorityEnum::HIGH, $mainChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, true), $selectionList->addEntityList(QueuePriorityEnum::HIGH, $priorityEntityIdsList); $mainChunkSize, ); $normalEntityIdsList = []; $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); $normalEntityChunkSize = $mainChunkSize - count($priorityEntityIdsList); if ($normalEntityChunkSize > 0) { if ($normalEntityChunkSize > 0) { $normalEntityIdsList = $this->extractIdsFromSet( $normalEntityIdsList = $this->extractIds(QueuePriorityEnum::NORMAL, $normalEntityChunkSize); AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, false), $selectionList->addEntityList(QueuePriorityEnum::NORMAL, $normalEntityIdsList); $normalEntityChunkSize ); } } return new SelectionList( return $selectionList; priorityEntityIdList: $priorityEntityIdsList, normalEntityIdList: $normalEntityIdsList, chunkSize: $mainChunkSize ); } } private function getStaleChunk(): array private function getStaleChunk(): array return []; return []; } } private function extractIds(QueuePriorityEnum $priority, int $chunkSize): array { return $this->extractIdsFromSet( AsyncUpdateElasticSearch::getEntitiesSourceList($this->updateTarget, $priority), $chunkSize ); } } } app/Component/ES/AsyncUpdateElasticSearch.php +17-10Lines changed: 17 additions & 10 deletions Original file line numberOriginal file lineDiff line numberDiff line change * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * Various methods and actions call this method to rebuild auto-score, plays, shares, topic triggers * and other activity related data * and other activity related data */ */ public static function markEntityForUpdate(string $entityType, int $entityId, bool $withPriority = false): void public static function markEntityForUpdate( { string $entityType, $targetList = self::selectTargetList($entityType, $withPriority); int $entityId, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { $targetList = self::selectTargetList($entityType, $priority); Redis::sadd($targetList, $entityId); Redis::sadd($targetList, $entityId); Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ Log::info('[ AsyncUpdateElasticSearch ] Entity added to Redis list', [ 'entityType' => $entityType, 'entityType' => $entityType, 'entityId' => $entityId, 'entityId' => $entityId, 'collectionKey' => $targetList, 'collectionKey' => $targetList, 'withPriority' => $withPriority, 'priority' => $priority->value, ]); ]); } } public static function bulkMarkEntitiesForUpdate( public static function bulkMarkEntitiesForUpdate( string $entityType, string $entityType, array $entityIds, array $entityIds, bool $withPriority = false QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): void { ): void { if (empty($entityIds)) { if (empty($entityIds)) { return; return; } } $targetList = self::selectTargetList($entityType, $withPriority); $targetList = self::selectTargetList($entityType, $priority); Redis::saddarray($targetList, $entityIds); Redis::saddarray($targetList, $entityIds); } } * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Expose the Redis list name, which holds the collection of scheduled entities to be processed. * Takes priority into consideration. * Takes priority into consideration. */ */ public static function getEntitiesSourceList(string $updateTarget, $priority = false): string public static function getEntitiesSourceList( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): string { return self::selectTargetList($updateTarget, $priority); return self::selectTargetList($updateTarget, $priority); } } /** /** * Externally called helper methods * Externally called helper methods * - count and flush * - count and flush */ */ public static function getEntitiesListCount(string $updateTarget, $priority = false): int public static function getEntitiesListCount( { string $updateTarget, QueuePriorityEnum $priority = QueuePriorityEnum::NORMAL ): int { $targetList = self::selectTargetList($updateTarget, $priority); $targetList = self::selectTargetList($updateTarget, $priority); return Redis::scard($targetList); return Redis::scard($targetList); app/Component/ES/QueuePriorityEnum.php +40Lines changed: 40 additions & 0 deletions Original file line numberOriginal file lineDiff line numberDiff line change <?php declare(strict_types=1); namespace Jiminny\Component\ES; use Jiminny\Exceptions\InvalidArgumentException; enum QueuePriorityEnum: string { case HIGH = 'high'; case NORMAL = 'normal'; case LOW = 'low'; public static function tryFromString(string $value): self { if (! self::isValid($value)) { throw new InvalidArgumentException( sprintf('Invalid queue priority "%s" selected', $value) ); } return self::tryFrom($value); } public static function isValid(string $value): bool { return self::tryFrom($value) !== null; } public function isLow(): bool { return $this === self::LOW; } public function isHigh(): bool { return $this === self::HIGH; } } app/Component/ES/UpdateProcessManager.php +12-9Lines changed: 12 additions & 9 deletions Original file line numberOriginal file lineDiff line numberDiff line change if ($selectionList->isEmpty()) { if ($selectionList->isEmpty()) { return false; return false; } } /** /** * A stale chunk will always be smaller than a regular chunk. * A stale chunk will always be smaller than a regular chunk. * That doesn't mean there aren't any other scheduled entities for processing * That doesn't mean there aren't any other scheduled entities for processing private function rescheduleThrottledEntities(SelectionList $selectionList): void private function rescheduleThrottledEntities(SelectionList $selectionList): void { { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $groups = $selectionList->getEntityListGroups(); $this->updateTarget, foreach ($groups as $eachGroup) { $selectionList->getNormalPriorityEntityId() $groupEntityIds = $selectionList->getEntityListItemsByGroup($eachGroup); ); if (! empty($groupEntityIds)) { AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate( $this->updateTarget, entityType: $this->updateTarget, $selectionList->getHighPriorityEntityIds(), entityIds: $groupEntityIds, true priority: QueuePriorityEnum::tryFromString($eachGroup) ); ); } } } } } } app/Console/Commands/Elasticsearch/AsyncUpdateEsEntities.php +9-3Lines changed: 9 additions & 3 deletions Original file line numberOriginal file lineDiff line numberDiff line changeuse Illuminate\Support\Str; use Illuminate\Support\Str; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\AsyncUpdateElasticSearch; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Component\ES\UpdateProcessManager; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; Redis::set($gaugeLockName, true); Redis::set($gaugeLockName, true); Redis::expire($gaugeLockName, 60); Redis::expire($gaugeLockName, 60); $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::HIGH), ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update', $updateTarget), stat: sprintf('jiminny.%s-for-update', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::NORMAL), ); ); $this->gaugeStats( $this->gaugeStats( stat: sprintf('jiminny.%s-for-update-priority', $updateTarget), stat: sprintf('jiminny.%s-for-update-low', $updateTarget), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, true), value: AsyncUpdateElasticSearch::getEntitiesListCount($updateTarget, QueuePriorityEnum::LOW), ); ); } } app/Console/Commands/Elasticsearch/ResetAsyncElasticSearchCommand.php +5-1Lines changed: 5 additions & 1 deletion Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\EntityQueryBuilder; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\Traits\ValidateUpdateTargetTrait; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\Processor\UpdateTarget; use Jiminny\Component\ES\QueuePriorityEnum; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Component\ES\Repositories\EsResetRepositoryInterface; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Exceptions\InvalidArgumentException; use Jiminny\Traits\GracefullyStoppable; use Jiminny\Traits\GracefullyStoppable; private function getRemainingScheduledEntities(bool $printStatus = false): int private function getRemainingScheduledEntities(bool $printStatus = false): int { { $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingRegular = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount($this->updateTarget, true); $remainingPriority = AsyncUpdateElasticSearch::getEntitiesListCount( $this->updateTarget, QueuePriorityEnum::HIGH ); if ($printStatus) { if ($printStatus) { $this->line( $this->line(
You said
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changenamespace Jiminny\Component\ES\Listeners;
namespace Jiminny\Component\ES\Listeners;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Jiminny\Contracts\ES\Events\UpdateMultipleEntities;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
AsyncUpdateElasticSearch::bulkMarkEntitiesForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntities(),
$event->getEntities(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
}
}
app/Component/ES/Listeners/UpdateSingleTargetListener.php
+9-1Lines changed: 9 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\Actions\TrackElasticSearchOperations;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\AsyncUpdateElasticSearch;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\Processor\EntityQueryBuilder;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Jiminny\Contracts\ES\Events\UpdateSingleEntity;
use Psr\Log\LoggerInterface;
use Psr\Log\LoggerInterface;
'entityId' => $event->getEntityId(),
'entityId' => $event->getEntityId(),
]);
]);
/**
* @important:
* Synchronous event should eventually be deprecated.
* Instant write operations are blocking and should be used sparingly.
* It is much better in terms of performance to write 100 or 1000 documents at once,
* and have one blocking reindex operation, instead of writing 10 documents synchronously.
*/
if ($event->isSyncEvent()) {
if ($event->isSyncEvent()) {
$this->handleSyncEvent($event);
$this->handleSyncEvent($event);
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
$event->getUpdateTargetValue(),
$event->getUpdateTargetValue(),
$event->getEntityId(),
$event->getEntityId(),
$event->isPriority()
$event->isPriority() ? QueuePriorityEnum::HIGH : QueuePriorityEnum::NORMAL,
);
);
}
}
app/Component/ES/Processor/Actions/UpsertDocumentsAction.php
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberOriginal file lineDiff line numberDiff line changeuse Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\DTOs\SimpleCollection;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\ResponseEvaluator;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\Processor\Traits\UpdateTargetTrait;
use Jiminny\Component\ES\QueuePriorityEnum;
use Jiminny\Exceptions\InvalidDataException;
use Jiminny\Exceptions\InvalidDataException;
use Sentry\Laravel\Facade as Sentry;
use Sentry\Laravel\Facade as Sentry;
AsyncUpdateElasticSearch::markEntityForUpdate(
AsyncUpdateElasticSearch::markEntityForUpdate(
entityType: $this->updateTarget,
entityType: $this->updateTarget,
entityId: (int) $document->getId(),
entityId: (int) $document->getId(),
withPriority: true
priority: QueuePriorityEnum::HIGH,
);
);
}
}
app/Component/ES/Processor/DTOs/SelectionList.php
+54-17Lines changed: 54 additions & 17 deletions
Original file line numberOriginal file lineDiff line numberDiff line change
namespace Jiminny\Component\ES\Processor\DTOs;
namespace Jiminny\Component\ES\Processor\DTOs;
use Jiminny\Component\ES\QueuePriorityEnum;
/**
/**
* A very basic and simple collection of entity IDs.
* A very basic and simple collection of entity IDs.
*/
*/
class SelectionList
class SelectionList
{
{
public function __construct(
private const int DEFAULT_CHUNK = 200;
private readonly array $priorityEntityIdList = [],
private readonly array $normalEntityIdList = [],
private int $chunkSize = self::DEFAULT_CHUNK;
private readonly bool $isStaleChunkSelected = false,
private bool $isStaleChunkSelected = false;
private readonly int $chunkSize = 200,
private array $selectedEntityIds = [];
) {
// Do nothing
public function setChunkSize(int $chunkSize): void
{
$this->chunkSize = $chunkSize;
}
}
public function getAllIds(): array
public function markStale(): void
{
{
return array_merge($this->priorityEntityIdList, $this->normalEntityIdList);
$this->isStaleChunkSelected = true;
}
}
public function count(): int
public function addEntityList(string|QueuePriorityEnum $priority, array $entityIds): void
{
{
return count($this->priorityEntityIdList) + count($this->normalEntityIdList);
$priorityKey = $this->normalisePriority($priority);
if (! array_key_exists($priorityKey, $this->selectedEntityIds)) {
$this->selectedEntityIds[$priorityKey] = [];
}
$this->selectedEntityIds[$priorityKey] = array_merge($this->selectedEntityIds[$priorityKey], $entityIds);
}
}
public function isEmpty(): bool
/**
* @return string[]
*/
public function getEntityListGroups(): array
{
{
return empty($this->priorityEntityIdList) && empty($this->normalEntityIdList);
return array_keys($this->selectedEntityIds);
}
}
public function getHighPriorityEntityIds(): array
public function getEntityListItemsByGroup(string|QueuePriorityEnum $priority): array
{
{
return $this->priorityEntityIdList;
$priorityKey = $this->normalisePriority($priority);
return $this->selectedEntityIds[$priorityKey] ?? [];
}
}
public function getNormalPriorityEntityId(): array
public function getAllIds(): array
{
{
return $this->normalEntityIdList;
$allIds = [];
foreach ($this->selectedEntityIds as $entityIds) {
$allIds = array_merge($allIds, $entityIds);
}
return $allIds;
}
public function count(): int
{
return array_sum(
array_map('count', $this->selectedEntityIds)
);
}
public function isEmpty(): bool
{
return $this->count() === 0;
}
}
public function isStaleChunkSelected(): bool...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes#diff-005 github.com/jiminny/app/pull/12082/changes#diff-00566b80af55fe41e9ad53dabe5735494b207ecf2d8da23f7436862fa9cf0d0a...
|
55692
|
|
55640
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
PRO
PRO
Conversation with Gemini
Conversation with Gemini
Hi Lukas
Where should we start?
Where should we start?
🖼️ Create image, button, tap to use tool
🖼️ Create image
🎸 Create music, button, tap to use tool
🎸 Create music
Help me learn, button, tap to use tool
Help me learn
Boost my day, button, tap to use tool
Boost my day
Write anything, button, tap to use tool
Write anything
Enter a prompt for Gemini
encrypted
Enter a prompt for Gemini
encrypted
Open upload file menu
Tools
Open mode picker
Pro
Microphone
Send message
Summarize page
Summarize page
Skip to content
Skip to content
Open menu
Homepage (g then d)
jiminny
jiminny
app
app
Search or jump to…
Type
/
to search
Chat with Copilot
Open Copilot…
Create new...
All issues(g then i)
All pull requests
All repositories
You have unread notifications(g then n)
Open user navigation menu
Repository navigation
Repository navigation
Code
Code
Pull requests (31)
Pull requests
(
31
)
Agents
Agents
Actions
Actions
Wiki
Wiki
Security and quality (1)
Security and quality
(
1
)
Insights
Insights
Settings
Settings
Important update
Important update
On April 24 we'll start using GitHub Copilot interaction data for AI model training unless you opt out.
Review this update
Review this update
and manage your preferences in your
GitHub account settings
GitHub account settings
.
Dismiss banner
JY-20808 low priority indexing queue #12082 Edit title
JY-20808 low priority indexing queue
#
12082
Edit title
Preview
Preview
Awaiting approval
Awaiting approval
Code
Code
Open
Vasil-Jiminny
Vasil-Jiminny
wants to merge 36 commits into
master
master
from
JY-20808-low-priority-indexing-queue
JY-20808-low-priority-indexing-queue
Copy head branch name to clipboard
Lines changed: 432 additions & 159 deletions
Conversation (5)
Conversation
(
5
)
Commits (36)
Commits
(
36
)
Checks (3)
Checks
(
3
)
Files changed (18)
Files changed
(
18
)
Pull Request Toolbar
Pull Request Toolbar
Collapse file tree
All commits
All commits
0 of 18 files viewed
Submit review
Submit
review
Open diff view settings
Open overview panel
Open comments panel
(
0
)
Filter files…
Filter options
File tree
File tree
app
Component/ES
Listeners
UpdateMultipleTargetsListener.php
UpdateMultipleTargetsListener.php
UpdateSingleTargetListener.php
UpdateSingleTargetListener.php
Processor
Actions
UpsertDocumentsAction.php
UpsertDocumentsAction.php
DTOs
SelectionList.php
SelectionList.php
Traits
SelectEntityListTrait.php
SelectEntityListTrait.php
TargetEntitiesSelector.php
TargetEntitiesSelector.php
AsyncUpdateElasticSearch.php
AsyncUpdateElasticSearch.php
QueuePriorityEnum.php
QueuePriorityEnum.php
UpdateProcessManager.php
UpdateProcessManager.php
Console/Commands/Elasticsearch
AsyncUpdateEsEntities.php
AsyncUpdateEsEntities.php
ResetAsyncElasticSearchCommand.php
ResetAsyncElasticSearchCommand.php
tests/Unit/Component/ES
Processor
DTOs
SelectionListTest.php
SelectionListTest.php
Traits
SelectEntityListTraitTest.php
SelectEntityListTraitTest.php
BatchStatusManagerTest.php
BatchStatusManagerTest.php
TargetEntitiesSelectorTest.php
TargetEntitiesSelectorTest.php
AsyncUpdateElasticSearchTest.php
AsyncUpdateElasticSearchTest.php
QueuePriorityEnumTest.php
QueuePriorityEnumTest.php
UpdateProcessManagerTest.php
UpdateProcessManagerTest.php
Collapse file
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
Copy file name to clipboard
Expand all lines: app/Component/ES/Listeners/UpdateMultipleTargetsListener.php...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes
|
55640
|
|
55641
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
PRO
PRO
Conversation with Gemini
Conversation with Gemini
Copy prompt
Edit
You said I’m on page “<tabTitle>JY-20808 low priority indexing queue by Vasil-Jimi</tabTitle>” with “<selection>@@ -5,84 +5,140 @@5namespace Tests\Unit\Component\ES\Processor\DTOs;5namespace Tests\Unit\Component\ES\Processor\DTOs;667use Jiminny\Component\ES\Processor\DTOs\SelectionList;7use Jiminny\Component\ES\Processor\DTOs\SelectionList;8+use Jiminny\Component\ES\QueuePriorityEnum;8use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\TestCase;10use PHPUnit\Framework\TestCase;101111#[CoversClass(SelectionList::class)]12#[CoversClass(SelectionList::class)]12final class SelectionListTest extends TestCase13final class SelectionListTest extends TestCase13{14{14-public function testGetEntityIds(): void15+private const int CHUNK_SIZE = 100;16+17+public function testAddEntityListAndGetAllIds(): void18+ {19+$selectionList = new SelectionList();20+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);21+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);22+23+$this->assertSame([1, 2, 3, 4], $selectionList->getAllIds());24+ }25+26+public function testAddEntityListMergesMultipleCallsForSamePriority(): void27+ {28+$selectionList = new SelectionList();29+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);30+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [3, 4]);31+32+$this->assertSame([1, 2, 3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));33+ }34+35+public function testGetEntityListItemsByGroup(): void36+ {37+$selectionList = new SelectionList();38+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);39+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);40+$selectionList->addEntityList(QueuePriorityEnum::LOW, [5, 6]);41+42+$this->assertSame([1, 2], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));43+$this->assertSame([3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::NORMAL));44+$this->assertSame([5, 6], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::LOW));45+ }46+47+public function testGetEntityListItemsByGroupReturnsEmptyArrayForUnknownGroup(): void15 {48 {16-$priority = ['p1', 'p2'];49+$selectionList = new SelectionList();17-$normal = ['n1', 'n2'];50+18-$selectionList = new SelectionList($priority, $normal);51+$this->assertSame([], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));52+ }195320-$this->assertSame($priority, $selectionList->getHighPriorityEntityIds());54+public function testGetEntityListGroups(): void21-$this->assertSame($normal, $selectionList->getNormalPriorityEntityId());55+ {56+$selectionList = new SelectionList();57+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);58+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2]);225923-$this->assertSame(['p1', 'p2', 'n1', 'n2'], $selectionList->getAllIds());60+$this->assertSame(['high', 'normal'], $selectionList->getEntityListGroups());24 }61 }256226public function testCountReturnsTotalCount(): void63public function testCountReturnsTotalCount(): void27 {64 {28-$selectionList = new SelectionList(['p1'], ['n1', 'n2']);65+$selectionList = new SelectionList();66+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);67+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2, 3]);68+29$this->assertSame(3, $selectionList->count());69$this->assertSame(3, $selectionList->count());30 }70 }317132-public function testIsEmptyReturnsTrueWhenBothListsAreEmpty(): void72+public function testIsEmptyReturnsTrueWhenNoEntitiesAdded(): void33 {73 {34-$selectionList = new SelectionList([], []);74+$selectionList = new SelectionList();75+35$this->assertTrue($selectionList->isEmpty());76$this->assertTrue($selectionList->isEmpty());36 }77 }377838-public function testIsEmptyReturnsFalseWhenAnyListIsNotEmpty(): void79+public function testIsEmptyReturnsFalseWhenEntitiesAdded(): void39 {80 {40-$selectionList = new SelectionList(['p1'], []);81+$selectionList = new SelectionList();41-$this->assertFalse($selectionList->isEmpty());82+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);428343-$selectionList2 = new SelectionList([], ['n1']);84+$this->assertFalse($selectionList->isEmpty());44-$this->assertFalse($selectionList2->isEmpty());45 }85 }468647-public function testIsStaleChunkSelected(): void87+public function testMarkStaleAndIsStaleChunkSelected(): void48 {88 {49-$selectionList = new SelectionList([], [], true);89+$selectionList = new SelectionList();50-$this->assertTrue($selectionList->isStaleChunkSelected());90+91+$this->assertFalse($selectionList->isStaleChunkSelected());92+93+$selectionList->markStale();519452-$selectionList2 = new SelectionList([], [], false);95+$this->assertTrue($selectionList->isStaleChunkSelected());53-$this->assertFalse($selectionList2->isStaleChunkSelected());54 }96 }559756public function testIsFullChunk(): void98public function testIsFullChunk(): void57 {99 {58-$chunkSize = 200;100+$selectionList = new SelectionList();59-101+$selectionList->setChunkSize(self::CHUNK_SIZE);60-$fullList = range(1, $chunkSize);102+61-$selectionList = new SelectionList(103+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, self::CHUNK_SIZE));62- priorityEntityIdList: $fullList,104+63- normalEntityIdList: [],105+$this->assertTrue($selectionList->isFullChunk());64- chunkSize: $chunkSize,106+ }65- );107+108+public function testIsNotFullChunkWhenBelowChunkSize(): void109+ {110+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);111+112+$selectionList = new SelectionList();113+$selectionList->setChunkSize(self::CHUNK_SIZE);114+115+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range(1, $halfRange));116+117+$this->assertFalse($selectionList->isFullChunk());118+ }119+120+public function testIsFullChunkAcrossMultiplePriorityGroups(): void121+ {122+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);123+124+$selectionList = new SelectionList();125+$selectionList->setChunkSize(self::CHUNK_SIZE);126+127+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, $halfRange));128+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range($halfRange + 1, self::CHUNK_SIZE));129+130+$this->assertTrue($selectionList->isFullChunk());131+ }132+133+public function testSetChunkSizeAffectsIsFullChunk(): void134+ {135+$selectionList = new SelectionList();136+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [1, 2, 3]);137+138+$selectionList->setChunkSize(3);66$this->assertTrue($selectionList->isFullChunk());139$this->assertTrue($selectionList->isFullChunk());6714068-$halfRange = (int) ceil($chunkSize / 2);141+$selectionList->setChunkSize(50);69-142+$this->assertFalse($selectionList->isFullChunk());70-$notFullList = range(1, $halfRange);71-$selectionList2 = new SelectionList(72- priorityEntityIdList: $notFullList,73- normalEntityIdList: [],74- chunkSize: $chunkSize,75- );76-$this->assertFalse($selectionList2->isFullChunk());77-78-$priorityRange = range(1, $halfRange);79-$normalRange = range($halfRange + 1, $chunkSize);80-81-$selectionList3 = new SelectionList(82- priorityEntityIdList: $priorityRange,83- normalEntityIdList: $normalRange,84- chunkSize: $chunkSize,85- );86-$this->assertTrue($selectionList3->isFullChunk());87 }143 }88}144}</selection>” selected. Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
You said
I’m on page “<tabTitle>JY-20808 low priority indexing queue by Vasil-Jimi</tabTitle>” with “<selection>@@ -5,84 +5,140 @@5namespace Tests\Unit\Component\ES\Processor\DTOs;5namespace Tests\Unit\Component\ES\Processor\DTOs;667use Jiminny\Component\ES\Processor\DTOs\SelectionList;7use Jiminny\Component\ES\Processor\DTOs\SelectionList;8+use Jiminny\Component\ES\QueuePriorityEnum;8use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\TestCase;10use PHPUnit\Framework\TestCase;101111#[CoversClass(SelectionList::class)]12#[CoversClass(SelectionList::class)]12final class SelectionListTest extends TestCase13final class SelectionListTest extends TestCase13{14{14-public function testGetEntityIds(): void15+private const int CHUNK_SIZE = 100;16+17+public function testAddEntityListAndGetAllIds(): void18+ {19+$selectionList = new SelectionList();20+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);21+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);22+23+$this->assertSame([1, 2, 3, 4], $selectionList->getAllIds());24+ }25+26+public function testAddEntityListMergesMultipleCallsForSamePriority(): void27+ {28+$selectionList = new SelectionList();29+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);30+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [3, 4]);31+32+$this->assertSame([1, 2, 3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));33+ }34+35+public function testGetEntityListItemsByGroup(): void36+ {37+$selectionList = new SelectionList();38+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);39+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);40+$selectionList->addEntityList(QueuePriorityEnum::LOW, [5, 6]);41+42+$this->assertSame([1, 2], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));43+$this->assertSame([3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::NORMAL));44+$this->assertSame([5, 6], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::LOW));45+ }46+47+public function testGetEntityListItemsByGroupReturnsEmptyArrayForUnknownGroup(): void15 {48 {16-$priority = ['p1', 'p2'];49+$selectionList = new SelectionList();17-$normal = ['n1', 'n2'];50+18-$selectionList = new SelectionList($priority, $normal);51+$this->assertSame([], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));52+ }195320-$this->assertSame($priority, $selectionList->getHighPriorityEntityIds());54+public function testGetEntityListGroups(): void21-$this->assertSame($normal, $selectionList->getNormalPriorityEntityId());55+ {56+$selectionList = new SelectionList();57+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);58+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2]);225923-$this->assertSame(['p1', 'p2', 'n1', 'n2'], $selectionList->getAllIds());60+$this->assertSame(['high', 'normal'], $selectionList->getEntityListGroups());24 }61 }256226public function testCountReturnsTotalCount(): void63public function testCountReturnsTotalCount(): void27 {64 {28-$selectionList = new SelectionList(['p1'], ['n1', 'n2']);65+$selectionList = new SelectionList();66+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);67+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2, 3]);68+29$this->assertSame(3, $selectionList->count());69$this->assertSame(3, $selectionList->count());30 }70 }317132-public function testIsEmptyReturnsTrueWhenBothListsAreEmpty(): void72+public function testIsEmptyReturnsTrueWhenNoEntitiesAdded(): void33 {73 {34-$selectionList = new SelectionList([], []);74+$selectionList = new SelectionList();75+35$this->assertTrue($selectionList->isEmpty());76$this->assertTrue($selectionList->isEmpty());36 }77 }377838-public function testIsEmptyReturnsFalseWhenAnyListIsNotEmpty(): void79+public function testIsEmptyReturnsFalseWhenEntitiesAdded(): void39 {80 {40-$selectionList = new SelectionList(['p1'], []);81+$selectionList = new SelectionList();41-$this->assertFalse($selectionList->isEmpty());82+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);428343-$selectionList2 = new SelectionList([], ['n1']);84+$this->assertFalse($selectionList->isEmpty());44-$this->assertFalse($selectionList2->isEmpty());45 }85 }468647-public function testIsStaleChunkSelected(): void87+public function testMarkStaleAndIsStaleChunkSelected(): void48 {88 {49-$selectionList = new SelectionList([], [], true);89+$selectionList = new SelectionList();50-$this->assertTrue($selectionList->isStaleChunkSelected());90+91+$this->assertFalse($selectionList->isStaleChunkSelected());92+93+$selectionList->markStale();519452-$selectionList2 = new SelectionList([], [], false);95+$this->assertTrue($selectionList->isStaleChunkSelected());53-$this->assertFalse($selectionList2->isStaleChunkSelected());54 }96 }559756public function testIsFullChunk(): void98public function testIsFullChunk(): void57 {99 {58-$chunkSize = 200;100+$selectionList = new SelectionList();59-101+$selectionList->setChunkSize(self::CHUNK_SIZE);60-$fullList = range(1, $chunkSize);102+61-$selectionList = new SelectionList(103+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, self::CHUNK_SIZE));62- priorityEntityIdList: $fullList,104+63- normalEntityIdList: [],105+$this->assertTrue($selectionList->isFullChunk());64- chunkSize: $chunkSize,106+ }65- );107+108+public function testIsNotFullChunkWhenBelowChunkSize(): void109+ {110+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);111+112+$selectionList = new SelectionList();113+$selectionList->setChunkSize(self::CHUNK_SIZE);114+115+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range(1, $halfRange));116+117+$this->assertFalse($selectionList->isFullChunk());118+ }119+120+public function testIsFullChunkAcrossMultiplePriorityGroups(): void121+ {122+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);123+124+$selectionList = new SelectionList();125+$selectionList->setChunkSize(self::CHUNK_SIZE);126+127+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, $halfRange));128+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range($halfRange + 1, self::CHUNK_SIZE));129+130+$this->assertTrue($selectionList->isFullChunk());131+ }132+133+public function testSetChunkSizeAffectsIsFullChunk(): void134+ {135+$selectionList = new SelectionList();136+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [1, 2, 3]);137+138+$selectionList->setChunkSize(3);66$this->assertTrue($selectionList->isFullChunk());139$this->assertTrue($selectionList->isFullChunk());6714068-$halfRange = (int) ceil($chunkSize / 2);141+$selectionList->setChunkSize(50);69-142+$this->assertFalse($selectionList->isFullChunk());70-$notFullList = range(1, $halfRange);71-$selectionList2 = new SelectionList(72- priorityEntityIdList: $notFullList,73- normalEntityIdList: [],74- chunkSize: $chunkSize,75- );76-$this->assertFalse($selectionList2->isFullChunk());77-78-$priorityRange = range(1, $halfRange);79-$normalRange = range($halfRange + 1, $chunkSize);80-81-$selectionList3 = new SelectionList(82- priorityEntityIdList: $priorityRange,83- normalEntityIdList: $normalRange,84- chunkSize: $chunkSize,85- );86-$this->assertTrue($selectionList3->isFullChunk());87 }143 }88}144}</selection>” selected.
Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
Expand
Enter a prompt for Gemini
encrypted
Enter a prompt for Gemini
encrypted
Open upload file menu
Tools
Open mode picker
Pro
Stop response
Your Jiminny chats aren’t used to improve our models. Gemini is AI. It can make mistakes, so double check it.
Your privacy & Gemini Opens in a new window
Your privacy & Gemini
Opens in a new window
Gemini is typing
Summarize page
Summarize page
Skip to content
Skip to content
Open menu
Homepage (g then d)
jiminny
jiminny
app
app
Search or jump to…
Type
/
to search
Chat with Copilot
Open Copilot…
Create new...
All issues(g then i)
All pull requests
All repositories
You have unread notifications(g then n)
Open user navigation menu
Repository navigation
Repository navigation
Code
Code
Pull requests (31)
Pull requests
(
31
)
Agents
Agents
Actions
Actions
Wiki
Wiki
Security and quality (1)
Security and quality
(
1
)
Insights
Insights
Settings
Settings
Important update
Important update
On April 24 we'll start using GitHub Copilot interaction data for AI model training unless you opt out.
Review this update
Review this update
and manage your preferences in your
GitHub account settings
GitHub account settings
.
Dismiss banner
JY-20808 low priority indexing queue #12082 Edit title
JY-20808 low priority indexing queue
#
12082
Edit title
Preview
Preview
Awaiting approval
Awaiting approval
Code
Code
Open
Vasil-Jiminny
Vasil-Jiminny
wants to merge 36 commits into
master
master
from
JY-20808-low-priority-indexing-queue
JY-20808-low-priority-indexing-queue
Copy head branch name to clipboard
Lines changed: 432 additions & 159 deletions
Conversation (5)
Conversation
(
5
)
Commits (36)
Commits
(
36
)
Checks (3)
Checks
(
3
)
Files changed (18)
Files changed
(
18
)
Pull Request Toolbar
Pull Request Toolbar
Collapse file tree
All commits
All commits
0 of 18 files viewed
Submit review
Submit
review
Open diff view settings
Open overview panel
Open comments panel
(
0
)
Filter files…
Filter options
File tree
File tree
app
Component/ES
Listeners
UpdateMultipleTargetsListener.php
UpdateMultipleTargetsListener.php
UpdateSingleTargetListener.php
UpdateSingleTargetListener.php
Processor
Actions
UpsertDocumentsAction.php
UpsertDocumentsAction.php
DTOs
SelectionList.php
SelectionList.php
Traits
SelectEntityListTrait.php
SelectEntityListTrait.php
TargetEntitiesSelector.php
TargetEntitiesSelector.php
AsyncUpdateElasticSearch.php
AsyncUpdateElasticSearch.php
QueuePriorityEnum.php
QueuePriorityEnum.php
UpdateProcessManager.php
UpdateProcessManager.php
Console/Commands/Elasticsearch
AsyncUpdateEsEntities.php
AsyncUpdateEsEntities.php
ResetAsyncElasticSearchCommand.php
ResetAsyncElasticSearchCommand.php
tests/Unit/Component/ES
Processor
DTOs
SelectionListTest.php
SelectionListTest.php
Traits
SelectEntityListTraitTest.php
SelectEntityListTraitTest.php
BatchStatusManagerTest.php
BatchStatusManagerTest.php
TargetEntitiesSelectorTest.php
TargetEntitiesSelectorTest.php
AsyncUpdateElasticSearchTest.php
AsyncUpdateElasticSearchTest.php
QueuePriorityEnumTest.php
QueuePriorityEnumTest.php
UpdateProcessManagerTest.php
UpdateProcessManagerTest.php
Collapse file
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
Copy file name to clipboard
Expand all lines: app/Component/ES/Listeners/UpdateMultipleTargetsListener.php
Lines changed: 2 additions & 1 deletion
Not Viewed
Viewed
Comment on this file
More options
Original file line number
Original file line
Diff line number
Diff line change
@@ -5,6 +5,7 @@
5
namespace
Jiminny
\
Component
\
ES
\
Listeners
;
5
namespace
Jiminny
\
Component
\
ES
\
Listeners
;
6
6
7
use
Jiminny
\
Component
\
ES
\
AsyncUpdateElasticSearch
;
7
use
Jiminny
\
Component
\
ES
\
AsyncUpdateElasticSearch
;
8
+
use
Jiminny
\
Component
\
ES
\
QueuePriorityEnum
;
8
use
Jiminny
\
Contracts
\
ES
\
Events
\
UpdateMultipleEntities
;...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes
|
55641
|
|
55642
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Copy prompt
Edit
You said I’m on page “<tabTitle>JY-20808 low priority indexing queue by Vasil-Jimi</tabTitle>” with “<selection>@@ -5,84 +5,140 @@5namespace Tests\Unit\Component\ES\Processor\DTOs;5namespace Tests\Unit\Component\ES\Processor\DTOs;667use Jiminny\Component\ES\Processor\DTOs\SelectionList;7use Jiminny\Component\ES\Processor\DTOs\SelectionList;8+use Jiminny\Component\ES\QueuePriorityEnum;8use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\TestCase;10use PHPUnit\Framework\TestCase;101111#[CoversClass(SelectionList::class)]12#[CoversClass(SelectionList::class)]12final class SelectionListTest extends TestCase13final class SelectionListTest extends TestCase13{14{14-public function testGetEntityIds(): void15+private const int CHUNK_SIZE = 100;16+17+public function testAddEntityListAndGetAllIds(): void18+ {19+$selectionList = new SelectionList();20+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);21+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);22+23+$this->assertSame([1, 2, 3, 4], $selectionList->getAllIds());24+ }25+26+public function testAddEntityListMergesMultipleCallsForSamePriority(): void27+ {28+$selectionList = new SelectionList();29+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);30+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [3, 4]);31+32+$this->assertSame([1, 2, 3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));33+ }34+35+public function testGetEntityListItemsByGroup(): void36+ {37+$selectionList = new SelectionList();38+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);39+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);40+$selectionList->addEntityList(QueuePriorityEnum::LOW, [5, 6]);41+42+$this->assertSame([1, 2], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));43+$this->assertSame([3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::NORMAL));44+$this->assertSame([5, 6], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::LOW));45+ }46+47+public function testGetEntityListItemsByGroupReturnsEmptyArrayForUnknownGroup(): void15 {48 {16-$priority = ['p1', 'p2'];49+$selectionList = new SelectionList();17-$normal = ['n1', 'n2'];50+18-$selectionList = new SelectionList($priority, $normal);51+$this->assertSame([], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));52+ }195320-$this->assertSame($priority, $selectionList->getHighPriorityEntityIds());54+public function testGetEntityListGroups(): void21-$this->assertSame($normal, $selectionList->getNormalPriorityEntityId());55+ {56+$selectionList = new SelectionList();57+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);58+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2]);225923-$this->assertSame(['p1', 'p2', 'n1', 'n2'], $selectionList->getAllIds());60+$this->assertSame(['high', 'normal'], $selectionList->getEntityListGroups());24 }61 }256226public function testCountReturnsTotalCount(): void63public function testCountReturnsTotalCount(): void27 {64 {28-$selectionList = new SelectionList(['p1'], ['n1', 'n2']);65+$selectionList = new SelectionList();66+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);67+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2, 3]);68+29$this->assertSame(3, $selectionList->count());69$this->assertSame(3, $selectionList->count());30 }70 }317132-public function testIsEmptyReturnsTrueWhenBothListsAreEmpty(): void72+public function testIsEmptyReturnsTrueWhenNoEntitiesAdded(): void33 {73 {34-$selectionList = new SelectionList([], []);74+$selectionList = new SelectionList();75+35$this->assertTrue($selectionList->isEmpty());76$this->assertTrue($selectionList->isEmpty());36 }77 }377838-public function testIsEmptyReturnsFalseWhenAnyListIsNotEmpty(): void79+public function testIsEmptyReturnsFalseWhenEntitiesAdded(): void39 {80 {40-$selectionList = new SelectionList(['p1'], []);81+$selectionList = new SelectionList();41-$this->assertFalse($selectionList->isEmpty());82+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);428343-$selectionList2 = new SelectionList([], ['n1']);84+$this->assertFalse($selectionList->isEmpty());44-$this->assertFalse($selectionList2->isEmpty());45 }85 }468647-public function testIsStaleChunkSelected(): void87+public function testMarkStaleAndIsStaleChunkSelected(): void48 {88 {49-$selectionList = new SelectionList([], [], true);89+$selectionList = new SelectionList();50-$this->assertTrue($selectionList->isStaleChunkSelected());90+91+$this->assertFalse($selectionList->isStaleChunkSelected());92+93+$selectionList->markStale();519452-$selectionList2 = new SelectionList([], [], false);95+$this->assertTrue($selectionList->isStaleChunkSelected());53-$this->assertFalse($selectionList2->isStaleChunkSelected());54 }96 }559756public function testIsFullChunk(): void98public function testIsFullChunk(): void57 {99 {58-$chunkSize = 200;100+$selectionList = new SelectionList();59-101+$selectionList->setChunkSize(self::CHUNK_SIZE);60-$fullList = range(1, $chunkSize);102+61-$selectionList = new SelectionList(103+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, self::CHUNK_SIZE));62- priorityEntityIdList: $fullList,104+63- normalEntityIdList: [],105+$this->assertTrue($selectionList->isFullChunk());64- chunkSize: $chunkSize,106+ }65- );107+108+public function testIsNotFullChunkWhenBelowChunkSize(): void109+ {110+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);111+112+$selectionList = new SelectionList();113+$selectionList->setChunkSize(self::CHUNK_SIZE);114+115+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range(1, $halfRange));116+117+$this->assertFalse($selectionList->isFullChunk());118+ }119+120+public function testIsFullChunkAcrossMultiplePriorityGroups(): void121+ {122+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);123+124+$selectionList = new SelectionList();125+$selectionList->setChunkSize(self::CHUNK_SIZE);126+127+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, $halfRange));128+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range($halfRange + 1, self::CHUNK_SIZE));129+130+$this->assertTrue($selectionList->isFullChunk());131+ }132+133+public function testSetChunkSizeAffectsIsFullChunk(): void134+ {135+$selectionList = new SelectionList();136+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [1, 2, 3]);137+138+$selectionList->setChunkSize(3);66$this->assertTrue($selectionList->isFullChunk());139$this->assertTrue($selectionList->isFullChunk());6714068-$halfRange = (int) ceil($chunkSize / 2);141+$selectionList->setChunkSize(50);69-142+$this->assertFalse($selectionList->isFullChunk());70-$notFullList = range(1, $halfRange);71-$selectionList2 = new SelectionList(72- priorityEntityIdList: $notFullList,73- normalEntityIdList: [],74- chunkSize: $chunkSize,75- );76-$this->assertFalse($selectionList2->isFullChunk());77-78-$priorityRange = range(1, $halfRange);79-$normalRange = range($halfRange + 1, $chunkSize);80-81-$selectionList3 = new SelectionList(82- priorityEntityIdList: $priorityRange,83- normalEntityIdList: $normalRange,84- chunkSize: $chunkSize,85- );86-$this->assertTrue($selectionList3->isFullChunk());87 }143 }88}144}</selection>” selected. Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
You said
I’m on page “<tabTitle>JY-20808 low priority indexing queue by Vasil-Jimi</tabTitle>” with “<selection>@@ -5,84 +5,140 @@5namespace Tests\Unit\Component\ES\Processor\DTOs;5namespace Tests\Unit\Component\ES\Processor\DTOs;667use Jiminny\Component\ES\Processor\DTOs\SelectionList;7use Jiminny\Component\ES\Processor\DTOs\SelectionList;8+use Jiminny\Component\ES\QueuePriorityEnum;8use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\TestCase;10use PHPUnit\Framework\TestCase;101111#[CoversClass(SelectionList::class)]12#[CoversClass(SelectionList::class)]12final class SelectionListTest extends TestCase13final class SelectionListTest extends TestCase13{14{14-public function testGetEntityIds(): void15+private const int CHUNK_SIZE = 100;16+17+public function testAddEntityListAndGetAllIds(): void18+ {19+$selectionList = new SelectionList();20+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);21+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);22+23+$this->assertSame([1, 2, 3, 4], $selectionList->getAllIds());24+ }25+26+public function testAddEntityListMergesMultipleCallsForSamePriority(): void27+ {28+$selectionList = new SelectionList();29+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);30+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [3, 4]);31+32+$this->assertSame([1, 2, 3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));33+ }34+35+public function testGetEntityListItemsByGroup(): void36+ {37+$selectionList = new SelectionList();38+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);39+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);40+$selectionList->addEntityList(QueuePriorityEnum::LOW, [5, 6]);41+42+$this->assertSame([1, 2], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));43+$this->assertSame([3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::NORMAL));44+$this->assertSame([5, 6], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::LOW));45+ }46+47+public function testGetEntityListItemsByGroupReturnsEmptyArrayForUnknownGroup(): void15 {48 {16-$priority = ['p1', 'p2'];49+$selectionList = new SelectionList();17-$normal = ['n1', 'n2'];50+18-$selectionList = new SelectionList($priority, $normal);51+$this->assertSame([], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));52+ }195320-$this->assertSame($priority, $selectionList->getHighPriorityEntityIds());54+public function testGetEntityListGroups(): void21-$this->assertSame($normal, $selectionList->getNormalPriorityEntityId());55+ {56+$selectionList = new SelectionList();57+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);58+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2]);225923-$this->assertSame(['p1', 'p2', 'n1', 'n2'], $selectionList->getAllIds());60+$this->assertSame(['high', 'normal'], $selectionList->getEntityListGroups());24 }61 }256226public function testCountReturnsTotalCount(): void63public function testCountReturnsTotalCount(): void27 {64 {28-$selectionList = new SelectionList(['p1'], ['n1', 'n2']);65+$selectionList = new SelectionList();66+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);67+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2, 3]);68+29$this->assertSame(3, $selectionList->count());69$this->assertSame(3, $selectionList->count());30 }70 }317132-public function testIsEmptyReturnsTrueWhenBothListsAreEmpty(): void72+public function testIsEmptyReturnsTrueWhenNoEntitiesAdded(): void33 {73 {34-$selectionList = new SelectionList([], []);74+$selectionList = new SelectionList();75+35$this->assertTrue($selectionList->isEmpty());76$this->assertTrue($selectionList->isEmpty());36 }77 }377838-public function testIsEmptyReturnsFalseWhenAnyListIsNotEmpty(): void79+public function testIsEmptyReturnsFalseWhenEntitiesAdded(): void39 {80 {40-$selectionList = new SelectionList(['p1'], []);81+$selectionList = new SelectionList();41-$this->assertFalse($selectionList->isEmpty());82+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);428343-$selectionList2 = new SelectionList([], ['n1']);84+$this->assertFalse($selectionList->isEmpty());44-$this->assertFalse($selectionList2->isEmpty());45 }85 }468647-public function testIsStaleChunkSelected(): void87+public function testMarkStaleAndIsStaleChunkSelected(): void48 {88 {49-$selectionList = new SelectionList([], [], true);89+$selectionList = new SelectionList();50-$this->assertTrue($selectionList->isStaleChunkSelected());90+91+$this->assertFalse($selectionList->isStaleChunkSelected());92+93+$selectionList->markStale();519452-$selectionList2 = new SelectionList([], [], false);95+$this->assertTrue($selectionList->isStaleChunkSelected());53-$this->assertFalse($selectionList2->isStaleChunkSelected());54 }96 }559756public function testIsFullChunk(): void98public function testIsFullChunk(): void57 {99 {58-$chunkSize = 200;100+$selectionList = new SelectionList();59-101+$selectionList->setChunkSize(self::CHUNK_SIZE);60-$fullList = range(1, $chunkSize);102+61-$selectionList = new SelectionList(103+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, self::CHUNK_SIZE));62- priorityEntityIdList: $fullList,104+63- normalEntityIdList: [],105+$this->assertTrue($selectionList->isFullChunk());64- chunkSize: $chunkSize,106+ }65- );107+108+public function testIsNotFullChunkWhenBelowChunkSize(): void109+ {110+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);111+112+$selectionList = new SelectionList();113+$selectionList->setChunkSize(self::CHUNK_SIZE);114+115+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range(1, $halfRange));116+117+$this->assertFalse($selectionList->isFullChunk());118+ }119+120+public function testIsFullChunkAcrossMultiplePriorityGroups(): void121+ {122+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);123+124+$selectionList = new SelectionList();125+$selectionList->setChunkSize(self::CHUNK_SIZE);126+127+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, $halfRange));128+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range($halfRange + 1, self::CHUNK_SIZE));129+130+$this->assertTrue($selectionList->isFullChunk());131+ }132+133+public function testSetChunkSizeAffectsIsFullChunk(): void134+ {135+$selectionList = new SelectionList();136+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [1, 2, 3]);137+138+$selectionList->setChunkSize(3);66$this->assertTrue($selectionList->isFullChunk());139$this->assertTrue($selectionList->isFullChunk());6714068-$halfRange = (int) ceil($chunkSize / 2);141+$selectionList->setChunkSize(50);69-142+$this->assertFalse($selectionList->isFullChunk());70-$notFullList = range(1, $halfRange);71-$selectionList2 = new SelectionList(72- priorityEntityIdList: $notFullList,73- normalEntityIdList: [],74- chunkSize: $chunkSize,75- );76-$this->assertFalse($selectionList2->isFullChunk());77-78-$priorityRange = range(1, $halfRange);79-$normalRange = range($halfRange + 1, $chunkSize);80-81-$selectionList3 = new SelectionList(82- priorityEntityIdList: $priorityRange,83- normalEntityIdList: $normalRange,84- chunkSize: $chunkSize,85- );86-$this->assertTrue($selectionList3->isFullChunk());87 }143 }88}144}</selection>” selected.
Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
Expand
Gemini said
Gemini said
Here is a precise and concise summary of the code selection,
which refactors
SelectionListTest
to accommodate a shift from fixed high/normal priority properties to a more dynamic,
enum-driven priority group architecture.
Refactored Priority Handling
Refactored Priority Handling
Dynamic Priority Groups:
Replaces fixed high/normal array parameters with the
QueuePriorityEnum
(
HIGH
,
NORMAL
,
LOW
),
allowing entity lists to be managed dynamically by group types.
Merged Registrations:
Validates that multiple
addEntityList()
calls targeting the same priority level properly merge their data instead of overwriting it.
Group Querying:
Tests new methods
getEntityListItemsByGroup()
and
getEntityListGroups()
to retrieve group-specific IDs or an array of active group names (e.
g.,
['high', 'normal']
).
Updated State and Count Validations
Updated State and Count Validations
Instantiation:
Tests now instantiate
SelectionList
without initial constructor arguments,
using setters and action methods instead.
Empty States:
Updates assertions (
isEmpty()
) to verify state based on whether any entities have been dynamically added,
rather than checking pre-defined constructor arrays.
Stale Flags:
Changes from passing a
stale
boolean via the constructor to explicitly triggering a mutable
$selectionList->markStale()
method.
Chunk Capacity Behavior
Chunk Capacity Behavior
Flexible Chunk Sizes:
Introduces a
setChunkSize()
method,
replacing the constructor-configured chunk boundary.
Capacity Assertions:
Retains and updates the logic for
isFullChunk()
,
ensuring it accurately evaluates capacity when elements span multiple dynamic priority groups or when the chunk limit is altered mid-lifecycle.
Enter a prompt for Gemini
encrypted
Enter a prompt for Gemini
encrypted
Open upload file menu
Tools
Open mode picker
Pro
Stop response
Your Jiminny chats aren’t used to improve our models. Gemini is AI. It can make mistakes, so double check it.
Your privacy & Gemini Opens in a new window
Your privacy & Gemini
Opens in a new window
Summarize page
Summarize page
Skip to content
Skip to content
Open menu
Homepage (g then d)
jiminny
jiminny
app
app
Search or jump to…
Type
/
to search
Chat with Copilot
Open Copilot…
Create new...
All issues(g then i)
All pull requests
All repositories
You have unread notifications(g then n)
Open user navigation menu
Repository navigation
Repository navigation
Code
Code
Pull requests (31)
Pull requests
(
31
)
Agents
Agents
Actions
Actions
Wiki
Wiki
Security and quality (1)
Security and quality
(
1
)
Insights
Insights
Settings
Settings
Important update
Important update
On April 24 we'll start using GitHub Copilot interaction data for AI model training unless you opt out.
Review this update
Review this update
and manage your preferences in your
GitHub account settings
GitHub account settings
.
Dismiss banner
JY-20808 low priority indexing queue #12082 Edit title
JY-20808 low priority indexing queue
#
12082
Edit title
Preview
Preview
Awaiting approval
Awaiting approval
Code
Code
Open
Vasil-Jiminny
Vasil-Jiminny
wants to merge 36 commits into
master
master
from
JY-20808-low-priority-indexing-queue
JY-20808-low-priority-indexing-queue
Copy head branch name to clipboard
Lines changed: 432 additions & 159 deletions
Conversation (5)
Conversation
(
5
)
Commits (36)
Commits
(
36
)
Checks (3)
Checks
(
3
)
Files changed (18)
Files changed
(
18
)
Pull Request Toolbar
Pull Request Toolbar
Collapse file tree
All commits
All commits
0 of 18 files viewed
Submit review
Submit
review
Open diff view settings
Open overview panel
Open comments panel
(
0
)
Filter files…
Filter options
File tree
File tree
app
Component/ES
Listeners
UpdateMultipleTargetsListener.php
UpdateMultipleTargetsListener.php
UpdateSingleTargetListener.php
UpdateSingleTargetListener.php
Processor
Actions
UpsertDocumentsAction.php
UpsertDocumentsAction.php
DTOs
SelectionList.php...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes
|
55642
|
|
55643
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Copy prompt
Edit
You said I’m on page “<tabTitle>JY-20808 low priority indexing queue by Vasil-Jimi</tabTitle>” with “<selection>@@ -5,84 +5,140 @@5namespace Tests\Unit\Component\ES\Processor\DTOs;5namespace Tests\Unit\Component\ES\Processor\DTOs;667use Jiminny\Component\ES\Processor\DTOs\SelectionList;7use Jiminny\Component\ES\Processor\DTOs\SelectionList;8+use Jiminny\Component\ES\QueuePriorityEnum;8use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\TestCase;10use PHPUnit\Framework\TestCase;101111#[CoversClass(SelectionList::class)]12#[CoversClass(SelectionList::class)]12final class SelectionListTest extends TestCase13final class SelectionListTest extends TestCase13{14{14-public function testGetEntityIds(): void15+private const int CHUNK_SIZE = 100;16+17+public function testAddEntityListAndGetAllIds(): void18+ {19+$selectionList = new SelectionList();20+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);21+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);22+23+$this->assertSame([1, 2, 3, 4], $selectionList->getAllIds());24+ }25+26+public function testAddEntityListMergesMultipleCallsForSamePriority(): void27+ {28+$selectionList = new SelectionList();29+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);30+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [3, 4]);31+32+$this->assertSame([1, 2, 3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));33+ }34+35+public function testGetEntityListItemsByGroup(): void36+ {37+$selectionList = new SelectionList();38+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);39+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);40+$selectionList->addEntityList(QueuePriorityEnum::LOW, [5, 6]);41+42+$this->assertSame([1, 2], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));43+$this->assertSame([3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::NORMAL));44+$this->assertSame([5, 6], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::LOW));45+ }46+47+public function testGetEntityListItemsByGroupReturnsEmptyArrayForUnknownGroup(): void15 {48 {16-$priority = ['p1', 'p2'];49+$selectionList = new SelectionList();17-$normal = ['n1', 'n2'];50+18-$selectionList = new SelectionList($priority, $normal);51+$this->assertSame([], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));52+ }195320-$this->assertSame($priority, $selectionList->getHighPriorityEntityIds());54+public function testGetEntityListGroups(): void21-$this->assertSame($normal, $selectionList->getNormalPriorityEntityId());55+ {56+$selectionList = new SelectionList();57+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);58+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2]);225923-$this->assertSame(['p1', 'p2', 'n1', 'n2'], $selectionList->getAllIds());60+$this->assertSame(['high', 'normal'], $selectionList->getEntityListGroups());24 }61 }256226public function testCountReturnsTotalCount(): void63public function testCountReturnsTotalCount(): void27 {64 {28-$selectionList = new SelectionList(['p1'], ['n1', 'n2']);65+$selectionList = new SelectionList();66+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);67+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2, 3]);68+29$this->assertSame(3, $selectionList->count());69$this->assertSame(3, $selectionList->count());30 }70 }317132-public function testIsEmptyReturnsTrueWhenBothListsAreEmpty(): void72+public function testIsEmptyReturnsTrueWhenNoEntitiesAdded(): void33 {73 {34-$selectionList = new SelectionList([], []);74+$selectionList = new SelectionList();75+35$this->assertTrue($selectionList->isEmpty());76$this->assertTrue($selectionList->isEmpty());36 }77 }377838-public function testIsEmptyReturnsFalseWhenAnyListIsNotEmpty(): void79+public function testIsEmptyReturnsFalseWhenEntitiesAdded(): void39 {80 {40-$selectionList = new SelectionList(['p1'], []);81+$selectionList = new SelectionList();41-$this->assertFalse($selectionList->isEmpty());82+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);428343-$selectionList2 = new SelectionList([], ['n1']);84+$this->assertFalse($selectionList->isEmpty());44-$this->assertFalse($selectionList2->isEmpty());45 }85 }468647-public function testIsStaleChunkSelected(): void87+public function testMarkStaleAndIsStaleChunkSelected(): void48 {88 {49-$selectionList = new SelectionList([], [], true);89+$selectionList = new SelectionList();50-$this->assertTrue($selectionList->isStaleChunkSelected());90+91+$this->assertFalse($selectionList->isStaleChunkSelected());92+93+$selectionList->markStale();519452-$selectionList2 = new SelectionList([], [], false);95+$this->assertTrue($selectionList->isStaleChunkSelected());53-$this->assertFalse($selectionList2->isStaleChunkSelected());54 }96 }559756public function testIsFullChunk(): void98public function testIsFullChunk(): void57 {99 {58-$chunkSize = 200;100+$selectionList = new SelectionList();59-101+$selectionList->setChunkSize(self::CHUNK_SIZE);60-$fullList = range(1, $chunkSize);102+61-$selectionList = new SelectionList(103+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, self::CHUNK_SIZE));62- priorityEntityIdList: $fullList,104+63- normalEntityIdList: [],105+$this->assertTrue($selectionList->isFullChunk());64- chunkSize: $chunkSize,106+ }65- );107+108+public function testIsNotFullChunkWhenBelowChunkSize(): void109+ {110+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);111+112+$selectionList = new SelectionList();113+$selectionList->setChunkSize(self::CHUNK_SIZE);114+115+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range(1, $halfRange));116+117+$this->assertFalse($selectionList->isFullChunk());118+ }119+120+public function testIsFullChunkAcrossMultiplePriorityGroups(): void121+ {122+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);123+124+$selectionList = new SelectionList();125+$selectionList->setChunkSize(self::CHUNK_SIZE);126+127+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, $halfRange));128+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range($halfRange + 1, self::CHUNK_SIZE));129+130+$this->assertTrue($selectionList->isFullChunk());131+ }132+133+public function testSetChunkSizeAffectsIsFullChunk(): void134+ {135+$selectionList = new SelectionList();136+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [1, 2, 3]);137+138+$selectionList->setChunkSize(3);66$this->assertTrue($selectionList->isFullChunk());139$this->assertTrue($selectionList->isFullChunk());6714068-$halfRange = (int) ceil($chunkSize / 2);141+$selectionList->setChunkSize(50);69-142+$this->assertFalse($selectionList->isFullChunk());70-$notFullList = range(1, $halfRange);71-$selectionList2 = new SelectionList(72- priorityEntityIdList: $notFullList,73- normalEntityIdList: [],74- chunkSize: $chunkSize,75- );76-$this->assertFalse($selectionList2->isFullChunk());77-78-$priorityRange = range(1, $halfRange);79-$normalRange = range($halfRange + 1, $chunkSize);80-81-$selectionList3 = new SelectionList(82- priorityEntityIdList: $priorityRange,83- normalEntityIdList: $normalRange,84- chunkSize: $chunkSize,85- );86-$this->assertTrue($selectionList3->isFullChunk());87 }143 }88}144}</selection>” selected. Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
You said
I’m on page “<tabTitle>JY-20808 low priority indexing queue by Vasil-Jimi</tabTitle>” with “<selection>@@ -5,84 +5,140 @@5namespace Tests\Unit\Component\ES\Processor\DTOs;5namespace Tests\Unit\Component\ES\Processor\DTOs;667use Jiminny\Component\ES\Processor\DTOs\SelectionList;7use Jiminny\Component\ES\Processor\DTOs\SelectionList;8+use Jiminny\Component\ES\QueuePriorityEnum;8use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\TestCase;10use PHPUnit\Framework\TestCase;101111#[CoversClass(SelectionList::class)]12#[CoversClass(SelectionList::class)]12final class SelectionListTest extends TestCase13final class SelectionListTest extends TestCase13{14{14-public function testGetEntityIds(): void15+private const int CHUNK_SIZE = 100;16+17+public function testAddEntityListAndGetAllIds(): void18+ {19+$selectionList = new SelectionList();20+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);21+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);22+23+$this->assertSame([1, 2, 3, 4], $selectionList->getAllIds());24+ }25+26+public function testAddEntityListMergesMultipleCallsForSamePriority(): void27+ {28+$selectionList = new SelectionList();29+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);30+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [3, 4]);31+32+$this->assertSame([1, 2, 3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));33+ }34+35+public function testGetEntityListItemsByGroup(): void36+ {37+$selectionList = new SelectionList();38+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);39+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);40+$selectionList->addEntityList(QueuePriorityEnum::LOW, [5, 6]);41+42+$this->assertSame([1, 2], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));43+$this->assertSame([3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::NORMAL));44+$this->assertSame([5, 6], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::LOW));45+ }46+47+public function testGetEntityListItemsByGroupReturnsEmptyArrayForUnknownGroup(): void15 {48 {16-$priority = ['p1', 'p2'];49+$selectionList = new SelectionList();17-$normal = ['n1', 'n2'];50+18-$selectionList = new SelectionList($priority, $normal);51+$this->assertSame([], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));52+ }195320-$this->assertSame($priority, $selectionList->getHighPriorityEntityIds());54+public function testGetEntityListGroups(): void21-$this->assertSame($normal, $selectionList->getNormalPriorityEntityId());55+ {56+$selectionList = new SelectionList();57+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);58+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2]);225923-$this->assertSame(['p1', 'p2', 'n1', 'n2'], $selectionList->getAllIds());60+$this->assertSame(['high', 'normal'], $selectionList->getEntityListGroups());24 }61 }256226public function testCountReturnsTotalCount(): void63public function testCountReturnsTotalCount(): void27 {64 {28-$selectionList = new SelectionList(['p1'], ['n1', 'n2']);65+$selectionList = new SelectionList();66+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);67+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2, 3]);68+29$this->assertSame(3, $selectionList->count());69$this->assertSame(3, $selectionList->count());30 }70 }317132-public function testIsEmptyReturnsTrueWhenBothListsAreEmpty(): void72+public function testIsEmptyReturnsTrueWhenNoEntitiesAdded(): void33 {73 {34-$selectionList = new SelectionList([], []);74+$selectionList = new SelectionList();75+35$this->assertTrue($selectionList->isEmpty());76$this->assertTrue($selectionList->isEmpty());36 }77 }377838-public function testIsEmptyReturnsFalseWhenAnyListIsNotEmpty(): void79+public function testIsEmptyReturnsFalseWhenEntitiesAdded(): void39 {80 {40-$selectionList = new SelectionList(['p1'], []);81+$selectionList = new SelectionList();41-$this->assertFalse($selectionList->isEmpty());82+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);428343-$selectionList2 = new SelectionList([], ['n1']);84+$this->assertFalse($selectionList->isEmpty());44-$this->assertFalse($selectionList2->isEmpty());45 }85 }468647-public function testIsStaleChunkSelected(): void87+public function testMarkStaleAndIsStaleChunkSelected(): void48 {88 {49-$selectionList = new SelectionList([], [], true);89+$selectionList = new SelectionList();50-$this->assertTrue($selectionList->isStaleChunkSelected());90+91+$this->assertFalse($selectionList->isStaleChunkSelected());92+93+$selectionList->markStale();519452-$selectionList2 = new SelectionList([], [], false);95+$this->assertTrue($selectionList->isStaleChunkSelected());53-$this->assertFalse($selectionList2->isStaleChunkSelected());54 }96 }559756public function testIsFullChunk(): void98public function testIsFullChunk(): void57 {99 {58-$chunkSize = 200;100+$selectionList = new SelectionList();59-101+$selectionList->setChunkSize(self::CHUNK_SIZE);60-$fullList = range(1, $chunkSize);102+61-$selectionList = new SelectionList(103+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, self::CHUNK_SIZE));62- priorityEntityIdList: $fullList,104+63- normalEntityIdList: [],105+$this->assertTrue($selectionList->isFullChunk());64- chunkSize: $chunkSize,106+ }65- );107+108+public function testIsNotFullChunkWhenBelowChunkSize(): void109+ {110+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);111+112+$selectionList = new SelectionList();113+$selectionList->setChunkSize(self::CHUNK_SIZE);114+115+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range(1, $halfRange));116+117+$this->assertFalse($selectionList->isFullChunk());118+ }119+120+public function testIsFullChunkAcrossMultiplePriorityGroups(): void121+ {122+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);123+124+$selectionList = new SelectionList();125+$selectionList->setChunkSize(self::CHUNK_SIZE);126+127+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, $halfRange));128+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range($halfRange + 1, self::CHUNK_SIZE));129+130+$this->assertTrue($selectionList->isFullChunk());131+ }132+133+public function testSetChunkSizeAffectsIsFullChunk(): void134+ {135+$selectionList = new SelectionList();136+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [1, 2, 3]);137+138+$selectionList->setChunkSize(3);66$this->assertTrue($selectionList->isFullChunk());139$this->assertTrue($selectionList->isFullChunk());6714068-$halfRange = (int) ceil($chunkSize / 2);141+$selectionList->setChunkSize(50);69-142+$this->assertFalse($selectionList->isFullChunk());70-$notFullList = range(1, $halfRange);71-$selectionList2 = new SelectionList(72- priorityEntityIdList: $notFullList,73- normalEntityIdList: [],74- chunkSize: $chunkSize,75- );76-$this->assertFalse($selectionList2->isFullChunk());77-78-$priorityRange = range(1, $halfRange);79-$normalRange = range($halfRange + 1, $chunkSize);80-81-$selectionList3 = new SelectionList(82- priorityEntityIdList: $priorityRange,83- normalEntityIdList: $normalRange,84- chunkSize: $chunkSize,85- );86-$this->assertTrue($selectionList3->isFullChunk());87 }143 }88}144}</selection>” selected.
Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
Expand
Listen
Gemini said
Gemini said
Here is a precise and concise summary of the code selection, which refactors
SelectionListTest
to accommodate a shift from fixed high/normal priority properties to a more dynamic, enum-driven priority group architecture.
Refactored Priority Handling
Refactored Priority Handling
Dynamic Priority Groups:
Replaces fixed high/normal array parameters with the
QueuePriorityEnum
(
HIGH
,
NORMAL
,
LOW
), allowing entity lists to be managed dynamically by group types.
Merged Registrations:
Validates that multiple
addEntityList()
calls targeting the same priority level properly merge their data instead of overwriting it.
Group Querying:
Tests new methods
getEntityListItemsByGroup()
and
getEntityListGroups()
to retrieve group-specific IDs or an array of active group names (e.g.,
['high', 'normal']
).
Updated State and Count Validations
Updated State and Count Validations
Instantiation:
Tests now instantiate
SelectionList
without initial constructor arguments, using setters and action methods instead.
Empty States:
Updates assertions (
isEmpty()
) to verify state based on whether any entities have been dynamically added, rather than checking pre-defined constructor arrays.
Stale Flags:
Changes from passing a
stale
boolean via the constructor to explicitly triggering a mutable
$selectionList->markStale()
method.
Chunk Capacity Behavior
Chunk Capacity Behavior
Flexible Chunk Sizes:
Introduces a
setChunkSize()
method, replacing the constructor-configured chunk boundary.
Capacity Assertions:
Retains and updates the logic for
isFullChunk()
, ensuring it accurately evaluates capacity when elements span multiple dynamic priority groups or when the chunk limit is altered mid-lifecycle.
Good response
Bad response
Redo
Share & export
Copy
Show more options
Enter a prompt for Gemini
encrypted
Enter a prompt for Gemini
encrypted
Open upload file menu
Tools
Open mode picker
Pro
Microphone
Send message
Your Jiminny chats aren’t used to improve our models. Gemini is AI. It can make mistakes, so double check it.
Your privacy & Gemini Opens in a new window
Your privacy & Gemini
Opens in a new window
Summarize page
Summarize page
Skip to content
Skip to content
Open menu
Homepage (g then d)
jiminny
jiminny
app
app
Search or jump to…
Type
/
to search
Chat with Copilot
Open Copilot…
Create new...
All issues(g then i)
All pull requests
All repositories
You have unread notifications(g then n)
Open user navigation menu
Repository navigation
Repository navigation
Code
Code
Pull requests (31)
Pull requests
(
31
)
Agents
Agents
Actions
Actions
Wiki
Wiki
Security and quality (1)
Security and quality
(
1
)
Insights
Insights
Settings
Settings
Important update
Important update
On April 24 we'll start using GitHub Copilot interaction data for AI model training unless you opt out.
Review this update
Review this update
and manage your preferences in your
GitHub account settings
GitHub account settings
.
Dismiss banner
JY-20808 low priority indexing queue #12082 Edit title
JY-20808 low priority indexing queue
#
12082
Edit title
Preview
Preview
Awaiting approval
Awaiting approval
Code
Code
Open
Vasil-Jiminny
Vasil-Jiminny
wants to merge 36 commits into
master
master
from
JY-20808-low-priority-indexing-queue
JY-20808-low-priority-indexing-queue
Copy head branch name to clipboard
Lines changed: 432 additions & 159 deletions
Conversation (5)
Conversation
(
5
)
Commits (36)
Commits
(
36
)
Checks (3)
Checks
(
3
)
Files changed (18)
Files changed
(
18
)
Pull Request Toolbar
Pull Request Toolbar
Collapse file tree
All commits
All commits
0 of 18 files viewed
Submit review
Submit
review
Open diff view settings
Open overview panel
Open comments panel
(
0
)
Filter files…
Filter options
File tree
File tree
app
Component/ES
Listeners
UpdateMultipleTargetsListener.php
UpdateMultipleTargetsListener.php
UpdateSingleTargetListener.php
UpdateSingleTargetListener.php
Processor
Actions
UpsertDocumentsAction.php
UpsertDocumentsAction.php
DTOs
SelectionList.php
SelectionList.php
Traits
SelectEntityListTrait.php
SelectEntityListTrait.php
TargetEntitiesSelector.php
TargetEntitiesSelector.php
AsyncUpdateElasticSearch.php
AsyncUpdateElasticSearch.php
QueuePriorityEnum.php
QueuePriorityEnum.php
UpdateProcessManager.php
UpdateProcessManager.php...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes
|
55643
|
|
55645
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Copy prompt
You said I’m on page “<tabTitle>JY-20808 low priority indexing queue by Vasil-Jimi</tabTitle>” with “<selection>@@ -5,84 +5,140 @@5namespace Tests\Unit\Component\ES\Processor\DTOs;5namespace Tests\Unit\Component\ES\Processor\DTOs;667use Jiminny\Component\ES\Processor\DTOs\SelectionList;7use Jiminny\Component\ES\Processor\DTOs\SelectionList;8+use Jiminny\Component\ES\QueuePriorityEnum;8use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\TestCase;10use PHPUnit\Framework\TestCase;101111#[CoversClass(SelectionList::class)]12#[CoversClass(SelectionList::class)]12final class SelectionListTest extends TestCase13final class SelectionListTest extends TestCase13{14{14-public function testGetEntityIds(): void15+private const int CHUNK_SIZE = 100;16+17+public function testAddEntityListAndGetAllIds(): void18+ {19+$selectionList = new SelectionList();20+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);21+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);22+23+$this->assertSame([1, 2, 3, 4], $selectionList->getAllIds());24+ }25+26+public function testAddEntityListMergesMultipleCallsForSamePriority(): void27+ {28+$selectionList = new SelectionList();29+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);30+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [3, 4]);31+32+$this->assertSame([1, 2, 3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));33+ }34+35+public function testGetEntityListItemsByGroup(): void36+ {37+$selectionList = new SelectionList();38+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);39+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);40+$selectionList->addEntityList(QueuePriorityEnum::LOW, [5, 6]);41+42+$this->assertSame([1, 2], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));43+$this->assertSame([3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::NORMAL));44+$this->assertSame([5, 6], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::LOW));45+ }46+47+public function testGetEntityListItemsByGroupReturnsEmptyArrayForUnknownGroup(): void15 {48 {16-$priority = ['p1', 'p2'];49+$selectionList = new SelectionList();17-$normal = ['n1', 'n2'];50+18-$selectionList = new SelectionList($priority, $normal);51+$this->assertSame([], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));52+ }195320-$this->assertSame($priority, $selectionList->getHighPriorityEntityIds());54+public function testGetEntityListGroups(): void21-$this->assertSame($normal, $selectionList->getNormalPriorityEntityId());55+ {56+$selectionList = new SelectionList();57+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);58+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2]);225923-$this->assertSame(['p1', 'p2', 'n1', 'n2'], $selectionList->getAllIds());60+$this->assertSame(['high', 'normal'], $selectionList->getEntityListGroups());24 }61 }256226public function testCountReturnsTotalCount(): void63public function testCountReturnsTotalCount(): void27 {64 {28-$selectionList = new SelectionList(['p1'], ['n1', 'n2']);65+$selectionList = new SelectionList();66+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);67+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2, 3]);68+29$this->assertSame(3, $selectionList->count());69$this->assertSame(3, $selectionList->count());30 }70 }317132-public function testIsEmptyReturnsTrueWhenBothListsAreEmpty(): void72+public function testIsEmptyReturnsTrueWhenNoEntitiesAdded(): void33 {73 {34-$selectionList = new SelectionList([], []);74+$selectionList = new SelectionList();75+35$this->assertTrue($selectionList->isEmpty());76$this->assertTrue($selectionList->isEmpty());36 }77 }377838-public function testIsEmptyReturnsFalseWhenAnyListIsNotEmpty(): void79+public function testIsEmptyReturnsFalseWhenEntitiesAdded(): void39 {80 {40-$selectionList = new SelectionList(['p1'], []);81+$selectionList = new SelectionList();41-$this->assertFalse($selectionList->isEmpty());82+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);428343-$selectionList2 = new SelectionList([], ['n1']);84+$this->assertFalse($selectionList->isEmpty());44-$this->assertFalse($selectionList2->isEmpty());45 }85 }468647-public function testIsStaleChunkSelected(): void87+public function testMarkStaleAndIsStaleChunkSelected(): void48 {88 {49-$selectionList = new SelectionList([], [], true);89+$selectionList = new SelectionList();50-$this->assertTrue($selectionList->isStaleChunkSelected());90+91+$this->assertFalse($selectionList->isStaleChunkSelected());92+93+$selectionList->markStale();519452-$selectionList2 = new SelectionList([], [], false);95+$this->assertTrue($selectionList->isStaleChunkSelected());53-$this->assertFalse($selectionList2->isStaleChunkSelected());54 }96 }559756public function testIsFullChunk(): void98public function testIsFullChunk(): void57 {99 {58-$chunkSize = 200;100+$selectionList = new SelectionList();59-101+$selectionList->setChunkSize(self::CHUNK_SIZE);60-$fullList = range(1, $chunkSize);102+61-$selectionList = new SelectionList(103+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, self::CHUNK_SIZE));62- priorityEntityIdList: $fullList,104+63- normalEntityIdList: [],105+$this->assertTrue($selectionList->isFullChunk());64- chunkSize: $chunkSize,106+ }65- );107+108+public function testIsNotFullChunkWhenBelowChunkSize(): void109+ {110+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);111+112+$selectionList = new SelectionList();113+$selectionList->setChunkSize(self::CHUNK_SIZE);114+115+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range(1, $halfRange));116+117+$this->assertFalse($selectionList->isFullChunk());118+ }119+120+public function testIsFullChunkAcrossMultiplePriorityGroups(): void121+ {122+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);123+124+$selectionList = new SelectionList();125+$selectionList->setChunkSize(self::CHUNK_SIZE);126+127+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, $halfRange));128+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range($halfRange + 1, self::CHUNK_SIZE));129+130+$this->assertTrue($selectionList->isFullChunk());131+ }132+133+public function testSetChunkSizeAffectsIsFullChunk(): void134+ {135+$selectionList = new SelectionList();136+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [1, 2, 3]);137+138+$selectionList->setChunkSize(3);66$this->assertTrue($selectionList->isFullChunk());139$this->assertTrue($selectionList->isFullChunk());6714068-$halfRange = (int) ceil($chunkSize / 2);141+$selectionList->setChunkSize(50);69-142+$this->assertFalse($selectionList->isFullChunk());70-$notFullList = range(1, $halfRange);71-$selectionList2 = new SelectionList(72- priorityEntityIdList: $notFullList,73- normalEntityIdList: [],74- chunkSize: $chunkSize,75- );76-$this->assertFalse($selectionList2->isFullChunk());77-78-$priorityRange = range(1, $halfRange);79-$normalRange = range($halfRange + 1, $chunkSize);80-81-$selectionList3 = new SelectionList(82- priorityEntityIdList: $priorityRange,83- normalEntityIdList: $normalRange,84- chunkSize: $chunkSize,85- );86-$this->assertTrue($selectionList3->isFullChunk());87 }143 }88}144}</selection>” selected. Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
You said
I’m on page “<tabTitle>JY-20808 low priority indexing queue by Vasil-Jimi</tabTitle>” with “<selection>@@ -5,84 +5,140 @@5namespace Tests\Unit\Component\ES\Processor\DTOs;5namespace Tests\Unit\Component\ES\Processor\DTOs;667use Jiminny\Component\ES\Processor\DTOs\SelectionList;7use Jiminny\Component\ES\Processor\DTOs\SelectionList;8+use Jiminny\Component\ES\QueuePriorityEnum;8use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\TestCase;10use PHPUnit\Framework\TestCase;101111#[CoversClass(SelectionList::class)]12#[CoversClass(SelectionList::class)]12final class SelectionListTest extends TestCase13final class SelectionListTest extends TestCase13{14{14-public function testGetEntityIds(): void15+private const int CHUNK_SIZE = 100;16+17+public function testAddEntityListAndGetAllIds(): void18+ {19+$selectionList = new SelectionList();20+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);21+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);22+23+$this->assertSame([1, 2, 3, 4], $selectionList->getAllIds());24+ }25+26+public function testAddEntityListMergesMultipleCallsForSamePriority(): void27+ {28+$selectionList = new SelectionList();29+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);30+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [3, 4]);31+32+$this->assertSame([1, 2, 3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));33+ }34+35+public function testGetEntityListItemsByGroup(): void36+ {37+$selectionList = new SelectionList();38+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);39+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);40+$selectionList->addEntityList(QueuePriorityEnum::LOW, [5, 6]);41+42+$this->assertSame([1, 2], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));43+$this->assertSame([3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::NORMAL));44+$this->assertSame([5, 6], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::LOW));45+ }46+47+public function testGetEntityListItemsByGroupReturnsEmptyArrayForUnknownGroup(): void15 {48 {16-$priority = ['p1', 'p2'];49+$selectionList = new SelectionList();17-$normal = ['n1', 'n2'];50+18-$selectionList = new SelectionList($priority, $normal);51+$this->assertSame([], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));52+ }195320-$this->assertSame($priority, $selectionList->getHighPriorityEntityIds());54+public function testGetEntityListGroups(): void21-$this->assertSame($normal, $selectionList->getNormalPriorityEntityId());55+ {56+$selectionList = new SelectionList();57+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);58+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2]);225923-$this->assertSame(['p1', 'p2', 'n1', 'n2'], $selectionList->getAllIds());60+$this->assertSame(['high', 'normal'], $selectionList->getEntityListGroups());24 }61 }256226public function testCountReturnsTotalCount(): void63public function testCountReturnsTotalCount(): void27 {64 {28-$selectionList = new SelectionList(['p1'], ['n1', 'n2']);65+$selectionList = new SelectionList();66+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);67+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2, 3]);68+29$this->assertSame(3, $selectionList->count());69$this->assertSame(3, $selectionList->count());30 }70 }317132-public function testIsEmptyReturnsTrueWhenBothListsAreEmpty(): void72+public function testIsEmptyReturnsTrueWhenNoEntitiesAdded(): void33 {73 {34-$selectionList = new SelectionList([], []);74+$selectionList = new SelectionList();75+35$this->assertTrue($selectionList->isEmpty());76$this->assertTrue($selectionList->isEmpty());36 }77 }377838-public function testIsEmptyReturnsFalseWhenAnyListIsNotEmpty(): void79+public function testIsEmptyReturnsFalseWhenEntitiesAdded(): void39 {80 {40-$selectionList = new SelectionList(['p1'], []);81+$selectionList = new SelectionList();41-$this->assertFalse($selectionList->isEmpty());82+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);428343-$selectionList2 = new SelectionList([], ['n1']);84+$this->assertFalse($selectionList->isEmpty());44-$this->assertFalse($selectionList2->isEmpty());45 }85 }468647-public function testIsStaleChunkSelected(): void87+public function testMarkStaleAndIsStaleChunkSelected(): void48 {88 {49-$selectionList = new SelectionList([], [], true);89+$selectionList = new SelectionList();50-$this->assertTrue($selectionList->isStaleChunkSelected());90+91+$this->assertFalse($selectionList->isStaleChunkSelected());92+93+$selectionList->markStale();519452-$selectionList2 = new SelectionList([], [], false);95+$this->assertTrue($selectionList->isStaleChunkSelected());53-$this->assertFalse($selectionList2->isStaleChunkSelected());54 }96 }559756public function testIsFullChunk(): void98public function testIsFullChunk(): void57 {99 {58-$chunkSize = 200;100+$selectionList = new SelectionList();59-101+$selectionList->setChunkSize(self::CHUNK_SIZE);60-$fullList = range(1, $chunkSize);102+61-$selectionList = new SelectionList(103+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, self::CHUNK_SIZE));62- priorityEntityIdList: $fullList,104+63- normalEntityIdList: [],105+$this->assertTrue($selectionList->isFullChunk());64- chunkSize: $chunkSize,106+ }65- );107+108+public function testIsNotFullChunkWhenBelowChunkSize(): void109+ {110+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);111+112+$selectionList = new SelectionList();113+$selectionList->setChunkSize(self::CHUNK_SIZE);114+115+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range(1, $halfRange));116+117+$this->assertFalse($selectionList->isFullChunk());118+ }119+120+public function testIsFullChunkAcrossMultiplePriorityGroups(): void121+ {122+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);123+124+$selectionList = new SelectionList();125+$selectionList->setChunkSize(self::CHUNK_SIZE);126+127+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, $halfRange));128+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range($halfRange + 1, self::CHUNK_SIZE));129+130+$this->assertTrue($selectionList->isFullChunk());131+ }132+133+public function testSetChunkSizeAffectsIsFullChunk(): void134+ {135+$selectionList = new SelectionList();136+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [1, 2, 3]);137+138+$selectionList->setChunkSize(3);66$this->assertTrue($selectionList->isFullChunk());139$this->assertTrue($selectionList->isFullChunk());6714068-$halfRange = (int) ceil($chunkSize / 2);141+$selectionList->setChunkSize(50);69-142+$this->assertFalse($selectionList->isFullChunk());70-$notFullList = range(1, $halfRange);71-$selectionList2 = new SelectionList(72- priorityEntityIdList: $notFullList,73- normalEntityIdList: [],74- chunkSize: $chunkSize,75- );76-$this->assertFalse($selectionList2->isFullChunk());77-78-$priorityRange = range(1, $halfRange);79-$normalRange = range($halfRange + 1, $chunkSize);80-81-$selectionList3 = new SelectionList(82- priorityEntityIdList: $priorityRange,83- normalEntityIdList: $normalRange,84- chunkSize: $chunkSize,85- );86-$this->assertTrue($selectionList3->isFullChunk());87 }143 }88}144}</selection>” selected.
Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
Expand
Listen
Gemini said
Gemini said
Here is a precise and concise summary of the code selection, which refactors
SelectionListTest
to accommodate a shift from fixed high/normal priority properties to a more dynamic, enum-driven priority group architecture.
Refactored Priority Handling
Refactored Priority Handling
Dynamic Priority Groups:
Replaces fixed high/normal array parameters with the
QueuePriorityEnum
(
HIGH
,
NORMAL
,
LOW
), allowing entity lists to be managed dynamically by group types.
Merged Registrations:
Validates that multiple
addEntityList()
calls targeting the same priority level properly merge their data instead of overwriting it.
Group Querying:
Tests new methods
getEntityListItemsByGroup()
and
getEntityListGroups()
to retrieve group-specific IDs or an array of active group names (e.g.,
['high', 'normal']
).
Updated State and Count Validations
Updated State and Count Validations
Instantiation:
Tests now instantiate
SelectionList
without initial constructor arguments, using setters and action methods instead.
Empty States:
Updates assertions (
isEmpty()
) to verify state based on whether any entities have been dynamically added, rather than checking pre-defined constructor arrays.
Stale Flags:
Changes from passing a
stale
boolean via the constructor to explicitly triggering a mutable
$selectionList->markStale()
method.
Chunk Capacity Behavior
Chunk Capacity Behavior
Flexible Chunk Sizes:
Introduces a
setChunkSize()
method, replacing the constructor-configured chunk boundary.
Capacity Assertions:
Retains and updates the logic for
isFullChunk()
, ensuring it accurately evaluates capacity when elements span multiple dynamic priority groups or when the chunk limit is altered mid-lifecycle.
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said Are there any issues or potential bolltenecks
You said
Are there any issues or potential bolltenecks
Listen
Show thinking
Gemini said
Gemini said
Based on the test diff you provided, we can infer a few potential issues and architectural bottlenecks in the underlying
SelectionList
implementation. While the tests are well-structured, the behavior they describe hints at a few areas you might want to keep an eye on, particularly for an Elasticsearch (ES) processor queue.
Here are the potential issues and bottlenecks to consider:
1. Array Merging Overhead (Performance)
1. Array Merging Overhead (Performance)
The
testAddEntityListMergesMultipleCallsForSamePriority
test shows that calling
addEntityList()
multiple times for the same priority group merges the arrays.
The Bottleneck:
If the underlying implementation uses
array_merge()
inside a loop to append these IDs, it creates a new array in memory every time it's called. If you are processing thousands of entities and appending them in small batches, this can lead to memory bloat and CPU degradation due to constant array reallocation.
Mitigation:
Ensure the implementation handles batch appends efficiently (e.g., using the spread operator
[...$existing, ...$new]
or appending via references) or groups the arrays before doing a single merge.
2. Lack of Uniqueness/Deduplication (Logic)
2. Lack of Uniqueness/Deduplication (Logic)
None of the tests explicitly verify what happens if duplicate IDs are added. For example, if you call
addEntityList(QueuePriorityEnum::HIGH, [1, 2])
and then
addEntityList(QueuePriorityEnum::HIGH, [2, 3])
.
The Issue:
Does the getter return
[1, 2, 2, 3]
or
[1, 2, 3]
? If duplicates are not filtered out, your ES processor might end up indexing the same entity multiple times in a single chunk, which wastes I/O and processor cycles.
Mitigation:
The underlying class should ideally enforce uniqueness (e.g., using
array_unique()
before resolving the chunk, or using ID keys as associative array keys to naturally prevent duplicates).
3. Memory Consumption with Large Chunks (Scalability)
3. Memory Consumption with Large Chunks (Scalability)
The tests introduce
setChunkSize()
. If this chunk size is configured to be very large (e.g., tens of thousands of IDs to bulk-send to Elasticsearch).
The Bottleneck:
Holding massive arrays of IDs in memory across different priority groups can quickly hit PHP memory limits, especially if this
SelectionList
object is kept alive during a long-running daemon or cron job.
Mitigation:
Monitor memory usage if
CHUNK_SIZE
grows, and ensure the
SelectionList
object is properly destroyed (or garbage collected) after the chunk is processed to free up memory.
4. Type Juggling
4. Type Juggling
In the original code, string IDs were used (
['p1', 'p2']
). In the refactored tests, integer IDs are used (
[1, 2]
).
The Issue:
If your system handles UUIDs (strings) vs. relational DB auto-increments (integers), make sure the underlying implementation uses strict comparisons (
===
) where necessary, especially if deduplicating arrays, to prevent PHP from juggling
'1'
and
1
improperly.
Would you like me to draft a quick test case to check how the class currently handles duplicate IDs, or is deduplication intentionally handled further down your pipeline?
Good response
Bad response
Redo
Share & export
Copy
Show more options
Enter a prompt for Gemini
encrypted
Enter a prompt for Gemini
encrypted
Open upload file menu
Tools
Open mode picker
Pro
Microphone
Send message
Your Jiminny chats aren’t used to improve our models. Gemini is AI. It can make mistakes, so double check it.
Your privacy & Gemini Opens in a new window
Your privacy & Gemini
Opens in a new window
Summarize page
Summarize page
Skip to content
Skip to content
Open menu
Homepage (g then d)
jiminny
jiminny
app
app
Search or jump to…
Type
/
to search
Chat with Copilot
Open Copilot…
Create new...
All issues(g then i)
All pull requests
All repositories
You have unread notifications(g then n)
Open user navigation menu
Repository navigation
Repository navigation
Code
Code
Pull requests (31)
Pull requests
(
31
)
Agents
Agents
Actions
Actions
Wiki
Wiki
Security and quality (1)
Security and quality
(
1
)
Insights
Insights
Settings
Settings
Important update
Important update
On April 24 we'll start using GitHub Copilot interaction data for AI model training unless you opt out.
Review this update
Review this update
and manage your preferences in your
GitHub account settings
GitHub account settings
.
Dismiss banner
JY-20808 low priority indexing queue #12082 Edit title
JY-20808 low priority indexing queue
#
12082
Edit title
Preview
Preview
Awaiting approval
Awaiting approval
Code
Code
Open
Vasil-Jiminny
Vasil-Jiminny...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes
|
55645
|
|
55646
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Copy prompt
You said I’m on page “<tabTitle>JY-20808 low priority indexing queue by Vasil-Jimi</tabTitle>” with “<selection>@@ -5,84 +5,140 @@5namespace Tests\Unit\Component\ES\Processor\DTOs;5namespace Tests\Unit\Component\ES\Processor\DTOs;667use Jiminny\Component\ES\Processor\DTOs\SelectionList;7use Jiminny\Component\ES\Processor\DTOs\SelectionList;8+use Jiminny\Component\ES\QueuePriorityEnum;8use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\TestCase;10use PHPUnit\Framework\TestCase;101111#[CoversClass(SelectionList::class)]12#[CoversClass(SelectionList::class)]12final class SelectionListTest extends TestCase13final class SelectionListTest extends TestCase13{14{14-public function testGetEntityIds(): void15+private const int CHUNK_SIZE = 100;16+17+public function testAddEntityListAndGetAllIds(): void18+ {19+$selectionList = new SelectionList();20+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);21+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);22+23+$this->assertSame([1, 2, 3, 4], $selectionList->getAllIds());24+ }25+26+public function testAddEntityListMergesMultipleCallsForSamePriority(): void27+ {28+$selectionList = new SelectionList();29+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);30+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [3, 4]);31+32+$this->assertSame([1, 2, 3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));33+ }34+35+public function testGetEntityListItemsByGroup(): void36+ {37+$selectionList = new SelectionList();38+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);39+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);40+$selectionList->addEntityList(QueuePriorityEnum::LOW, [5, 6]);41+42+$this->assertSame([1, 2], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));43+$this->assertSame([3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::NORMAL));44+$this->assertSame([5, 6], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::LOW));45+ }46+47+public function testGetEntityListItemsByGroupReturnsEmptyArrayForUnknownGroup(): void15 {48 {16-$priority = ['p1', 'p2'];49+$selectionList = new SelectionList();17-$normal = ['n1', 'n2'];50+18-$selectionList = new SelectionList($priority, $normal);51+$this->assertSame([], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));52+ }195320-$this->assertSame($priority, $selectionList->getHighPriorityEntityIds());54+public function testGetEntityListGroups(): void21-$this->assertSame($normal, $selectionList->getNormalPriorityEntityId());55+ {56+$selectionList = new SelectionList();57+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);58+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2]);225923-$this->assertSame(['p1', 'p2', 'n1', 'n2'], $selectionList->getAllIds());60+$this->assertSame(['high', 'normal'], $selectionList->getEntityListGroups());24 }61 }256226public function testCountReturnsTotalCount(): void63public function testCountReturnsTotalCount(): void27 {64 {28-$selectionList = new SelectionList(['p1'], ['n1', 'n2']);65+$selectionList = new SelectionList();66+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);67+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2, 3]);68+29$this->assertSame(3, $selectionList->count());69$this->assertSame(3, $selectionList->count());30 }70 }317132-public function testIsEmptyReturnsTrueWhenBothListsAreEmpty(): void72+public function testIsEmptyReturnsTrueWhenNoEntitiesAdded(): void33 {73 {34-$selectionList = new SelectionList([], []);74+$selectionList = new SelectionList();75+35$this->assertTrue($selectionList->isEmpty());76$this->assertTrue($selectionList->isEmpty());36 }77 }377838-public function testIsEmptyReturnsFalseWhenAnyListIsNotEmpty(): void79+public function testIsEmptyReturnsFalseWhenEntitiesAdded(): void39 {80 {40-$selectionList = new SelectionList(['p1'], []);81+$selectionList = new SelectionList();41-$this->assertFalse($selectionList->isEmpty());82+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);428343-$selectionList2 = new SelectionList([], ['n1']);84+$this->assertFalse($selectionList->isEmpty());44-$this->assertFalse($selectionList2->isEmpty());45 }85 }468647-public function testIsStaleChunkSelected(): void87+public function testMarkStaleAndIsStaleChunkSelected(): void48 {88 {49-$selectionList = new SelectionList([], [], true);89+$selectionList = new SelectionList();50-$this->assertTrue($selectionList->isStaleChunkSelected());90+91+$this->assertFalse($selectionList->isStaleChunkSelected());92+93+$selectionList->markStale();519452-$selectionList2 = new SelectionList([], [], false);95+$this->assertTrue($selectionList->isStaleChunkSelected());53-$this->assertFalse($selectionList2->isStaleChunkSelected());54 }96 }559756public function testIsFullChunk(): void98public function testIsFullChunk(): void57 {99 {58-$chunkSize = 200;100+$selectionList = new SelectionList();59-101+$selectionList->setChunkSize(self::CHUNK_SIZE);60-$fullList = range(1, $chunkSize);102+61-$selectionList = new SelectionList(103+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, self::CHUNK_SIZE));62- priorityEntityIdList: $fullList,104+63- normalEntityIdList: [],105+$this->assertTrue($selectionList->isFullChunk());64- chunkSize: $chunkSize,106+ }65- );107+108+public function testIsNotFullChunkWhenBelowChunkSize(): void109+ {110+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);111+112+$selectionList = new SelectionList();113+$selectionList->setChunkSize(self::CHUNK_SIZE);114+115+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range(1, $halfRange));116+117+$this->assertFalse($selectionList->isFullChunk());118+ }119+120+public function testIsFullChunkAcrossMultiplePriorityGroups(): void121+ {122+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);123+124+$selectionList = new SelectionList();125+$selectionList->setChunkSize(self::CHUNK_SIZE);126+127+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, $halfRange));128+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range($halfRange + 1, self::CHUNK_SIZE));129+130+$this->assertTrue($selectionList->isFullChunk());131+ }132+133+public function testSetChunkSizeAffectsIsFullChunk(): void134+ {135+$selectionList = new SelectionList();136+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [1, 2, 3]);137+138+$selectionList->setChunkSize(3);66$this->assertTrue($selectionList->isFullChunk());139$this->assertTrue($selectionList->isFullChunk());6714068-$halfRange = (int) ceil($chunkSize / 2);141+$selectionList->setChunkSize(50);69-142+$this->assertFalse($selectionList->isFullChunk());70-$notFullList = range(1, $halfRange);71-$selectionList2 = new SelectionList(72- priorityEntityIdList: $notFullList,73- normalEntityIdList: [],74- chunkSize: $chunkSize,75- );76-$this->assertFalse($selectionList2->isFullChunk());77-78-$priorityRange = range(1, $halfRange);79-$normalRange = range($halfRange + 1, $chunkSize);80-81-$selectionList3 = new SelectionList(82- priorityEntityIdList: $priorityRange,83- normalEntityIdList: $normalRange,84- chunkSize: $chunkSize,85- );86-$this->assertTrue($selectionList3->isFullChunk());87 }143 }88}144}</selection>” selected. Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
You said
I’m on page “<tabTitle>JY-20808 low priority indexing queue by Vasil-Jimi</tabTitle>” with “<selection>@@ -5,84 +5,140 @@5namespace Tests\Unit\Component\ES\Processor\DTOs;5namespace Tests\Unit\Component\ES\Processor\DTOs;667use Jiminny\Component\ES\Processor\DTOs\SelectionList;7use Jiminny\Component\ES\Processor\DTOs\SelectionList;8+use Jiminny\Component\ES\QueuePriorityEnum;8use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\TestCase;10use PHPUnit\Framework\TestCase;101111#[CoversClass(SelectionList::class)]12#[CoversClass(SelectionList::class)]12final class SelectionListTest extends TestCase13final class SelectionListTest extends TestCase13{14{14-public function testGetEntityIds(): void15+private const int CHUNK_SIZE = 100;16+17+public function testAddEntityListAndGetAllIds(): void18+ {19+$selectionList = new SelectionList();20+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);21+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);22+23+$this->assertSame([1, 2, 3, 4], $selectionList->getAllIds());24+ }25+26+public function testAddEntityListMergesMultipleCallsForSamePriority(): void27+ {28+$selectionList = new SelectionList();29+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);30+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [3, 4]);31+32+$this->assertSame([1, 2, 3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));33+ }34+35+public function testGetEntityListItemsByGroup(): void36+ {37+$selectionList = new SelectionList();38+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);39+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);40+$selectionList->addEntityList(QueuePriorityEnum::LOW, [5, 6]);41+42+$this->assertSame([1, 2], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));43+$this->assertSame([3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::NORMAL));44+$this->assertSame([5, 6], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::LOW));45+ }46+47+public function testGetEntityListItemsByGroupReturnsEmptyArrayForUnknownGroup(): void15 {48 {16-$priority = ['p1', 'p2'];49+$selectionList = new SelectionList();17-$normal = ['n1', 'n2'];50+18-$selectionList = new SelectionList($priority, $normal);51+$this->assertSame([], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));52+ }195320-$this->assertSame($priority, $selectionList->getHighPriorityEntityIds());54+public function testGetEntityListGroups(): void21-$this->assertSame($normal, $selectionList->getNormalPriorityEntityId());55+ {56+$selectionList = new SelectionList();57+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);58+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2]);225923-$this->assertSame(['p1', 'p2', 'n1', 'n2'], $selectionList->getAllIds());60+$this->assertSame(['high', 'normal'], $selectionList->getEntityListGroups());24 }61 }256226public function testCountReturnsTotalCount(): void63public function testCountReturnsTotalCount(): void27 {64 {28-$selectionList = new SelectionList(['p1'], ['n1', 'n2']);65+$selectionList = new SelectionList();66+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);67+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2, 3]);68+29$this->assertSame(3, $selectionList->count());69$this->assertSame(3, $selectionList->count());30 }70 }317132-public function testIsEmptyReturnsTrueWhenBothListsAreEmpty(): void72+public function testIsEmptyReturnsTrueWhenNoEntitiesAdded(): void33 {73 {34-$selectionList = new SelectionList([], []);74+$selectionList = new SelectionList();75+35$this->assertTrue($selectionList->isEmpty());76$this->assertTrue($selectionList->isEmpty());36 }77 }377838-public function testIsEmptyReturnsFalseWhenAnyListIsNotEmpty(): void79+public function testIsEmptyReturnsFalseWhenEntitiesAdded(): void39 {80 {40-$selectionList = new SelectionList(['p1'], []);81+$selectionList = new SelectionList();41-$this->assertFalse($selectionList->isEmpty());82+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);428343-$selectionList2 = new SelectionList([], ['n1']);84+$this->assertFalse($selectionList->isEmpty());44-$this->assertFalse($selectionList2->isEmpty());45 }85 }468647-public function testIsStaleChunkSelected(): void87+public function testMarkStaleAndIsStaleChunkSelected(): void48 {88 {49-$selectionList = new SelectionList([], [], true);89+$selectionList = new SelectionList();50-$this->assertTrue($selectionList->isStaleChunkSelected());90+91+$this->assertFalse($selectionList->isStaleChunkSelected());92+93+$selectionList->markStale();519452-$selectionList2 = new SelectionList([], [], false);95+$this->assertTrue($selectionList->isStaleChunkSelected());53-$this->assertFalse($selectionList2->isStaleChunkSelected());54 }96 }559756public function testIsFullChunk(): void98public function testIsFullChunk(): void57 {99 {58-$chunkSize = 200;100+$selectionList = new SelectionList();59-101+$selectionList->setChunkSize(self::CHUNK_SIZE);60-$fullList = range(1, $chunkSize);102+61-$selectionList = new SelectionList(103+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, self::CHUNK_SIZE));62- priorityEntityIdList: $fullList,104+63- normalEntityIdList: [],105+$this->assertTrue($selectionList->isFullChunk());64- chunkSize: $chunkSize,106+ }65- );107+108+public function testIsNotFullChunkWhenBelowChunkSize(): void109+ {110+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);111+112+$selectionList = new SelectionList();113+$selectionList->setChunkSize(self::CHUNK_SIZE);114+115+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range(1, $halfRange));116+117+$this->assertFalse($selectionList->isFullChunk());118+ }119+120+public function testIsFullChunkAcrossMultiplePriorityGroups(): void121+ {122+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);123+124+$selectionList = new SelectionList();125+$selectionList->setChunkSize(self::CHUNK_SIZE);126+127+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, $halfRange));128+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range($halfRange + 1, self::CHUNK_SIZE));129+130+$this->assertTrue($selectionList->isFullChunk());131+ }132+133+public function testSetChunkSizeAffectsIsFullChunk(): void134+ {135+$selectionList = new SelectionList();136+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [1, 2, 3]);137+138+$selectionList->setChunkSize(3);66$this->assertTrue($selectionList->isFullChunk());139$this->assertTrue($selectionList->isFullChunk());6714068-$halfRange = (int) ceil($chunkSize / 2);141+$selectionList->setChunkSize(50);69-142+$this->assertFalse($selectionList->isFullChunk());70-$notFullList = range(1, $halfRange);71-$selectionList2 = new SelectionList(72- priorityEntityIdList: $notFullList,73- normalEntityIdList: [],74- chunkSize: $chunkSize,75- );76-$this->assertFalse($selectionList2->isFullChunk());77-78-$priorityRange = range(1, $halfRange);79-$normalRange = range($halfRange + 1, $chunkSize);80-81-$selectionList3 = new SelectionList(82- priorityEntityIdList: $priorityRange,83- normalEntityIdList: $normalRange,84- chunkSize: $chunkSize,85- );86-$this->assertTrue($selectionList3->isFullChunk());87 }143 }88}144}</selection>” selected.
Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
Expand
Listen
Gemini said
Gemini said
Here is a precise and concise summary of the code selection, which refactors
SelectionListTest
to accommodate a shift from fixed high/normal priority properties to a more dynamic, enum-driven priority group architecture.
Refactored Priority Handling
Refactored Priority Handling
Dynamic Priority Groups:
Replaces fixed high/normal array parameters with the
QueuePriorityEnum
(
HIGH
,
NORMAL
,
LOW
), allowing entity lists to be managed dynamically by group types.
Merged Registrations:
Validates that multiple
addEntityList()
calls targeting the same priority level properly merge their data instead of overwriting it.
Group Querying:
Tests new methods
getEntityListItemsByGroup()
and
getEntityListGroups()
to retrieve group-specific IDs or an array of active group names (e.g.,
['high', 'normal']
).
Updated State and Count Validations
Updated State and Count Validations
Instantiation:
Tests now instantiate
SelectionList
without initial constructor arguments, using setters and action methods instead.
Empty States:
Updates assertions (
isEmpty()
) to verify state based on whether any entities have been dynamically added, rather than checking pre-defined constructor arrays.
Stale Flags:
Changes from passing a
stale
boolean via the constructor to explicitly triggering a mutable
$selectionList->markStale()
method.
Chunk Capacity Behavior
Chunk Capacity Behavior
Flexible Chunk Sizes:
Introduces a
setChunkSize()
method, replacing the constructor-configured chunk boundary.
Capacity Assertions:
Retains and updates the logic for
isFullChunk()
, ensuring it accurately evaluates capacity when elements span multiple dynamic priority groups or when the chunk limit is altered mid-lifecycle.
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said Are there any issues or potential bolltenecks
You said
Are there any issues or potential bolltenecks
Listen
Show thinking
Gemini said
Gemini said
Based on the test diff you provided, we can infer a few potential issues and architectural bottlenecks in the underlying
SelectionList
implementation. While the tests are well-structured, the behavior they describe hints at a few areas you might want to keep an eye on, particularly for an Elasticsearch (ES) processor queue.
Here are the potential issues and bottlenecks to consider:
1. Array Merging Overhead (Performance)
1. Array Merging Overhead (Performance)
The
testAddEntityListMergesMultipleCallsForSamePriority
test shows that calling
addEntityList()
multiple times for the same priority group merges the arrays.
The Bottleneck:
If the underlying implementation uses
array_merge()
inside a loop to append these IDs, it creates a new array in memory every time it's called. If you are processing thousands of entities and appending them in small batches, this can lead to memory bloat and CPU degradation due to constant array reallocation.
Mitigation:
Ensure the implementation handles batch appends efficiently (e.g., using the spread operator
[...$existing, ...$new]
or appending via references) or groups the arrays before doing a single merge.
2. Lack of Uniqueness/Deduplication (Logic)
2. Lack of Uniqueness/Deduplication (Logic)
None of the tests explicitly verify what happens if duplicate IDs are added. For example, if you call
addEntityList(QueuePriorityEnum::HIGH, [1, 2])
and then
addEntityList(QueuePriorityEnum::HIGH, [2, 3])
.
The Issue:
Does the getter return
[1, 2, 2, 3]
or
[1, 2, 3]
? If duplicates are not filtered out, your ES processor might end up indexing the same entity multiple times in a single chunk, which wastes I/O and processor cycles.
Mitigation:
The underlying class should ideally enforce uniqueness (e.g., using
array_unique()
before resolving the chunk, or using ID keys as associative array keys to naturally prevent duplicates).
3. Memory Consumption with Large Chunks (Scalability)
3. Memory Consumption with Large Chunks (Scalability)
The tests introduce
setChunkSize()
. If this chunk size is configured to be very large (e.g., tens of thousands of IDs to bulk-send to Elasticsearch).
The Bottleneck:
Holding massive arrays of IDs in memory across different priority groups can quickly hit PHP memory limits, especially if this
SelectionList
object is kept alive during a long-running daemon or cron job.
Mitigation:
Monitor memory usage if
CHUNK_SIZE
grows, and ensure the
SelectionList
object is properly destroyed (or garbage collected) after the chunk is processed to free up memory.
4. Type Juggling
4. Type Juggling
In the original code, string IDs were used (
['p1', 'p2']
). In the refactored tests, integer IDs are used (
[1, 2]
).
The Issue:
If your system handles UUIDs (strings) vs. relational DB auto-increments (integers), make sure the underlying implementation uses strict comparisons (
===
) where necessary, especially if deduplicating arrays, to prevent PHP from juggling
'1'
and
1
improperly.
Would you like me to draft a quick test case to check how the class currently handles duplicate IDs, or is deduplication intentionally handled further down your pipeline?
Good response
Bad response
Redo
Share & export
Copy
Show more options
Enter a prompt for Gemini
encrypted
Enter a prompt for Gemini
encrypted
Open upload file menu
Tools
Open mode picker
Pro
Microphone...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes
|
55646
|
|
55647
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Copy prompt
You said I’m on page “<tabTitle>JY-20808 low priority indexing queue by Vasil-Jimi</tabTitle>” with “<selection>@@ -5,84 +5,140 @@5namespace Tests\Unit\Component\ES\Processor\DTOs;5namespace Tests\Unit\Component\ES\Processor\DTOs;667use Jiminny\Component\ES\Processor\DTOs\SelectionList;7use Jiminny\Component\ES\Processor\DTOs\SelectionList;8+use Jiminny\Component\ES\QueuePriorityEnum;8use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\TestCase;10use PHPUnit\Framework\TestCase;101111#[CoversClass(SelectionList::class)]12#[CoversClass(SelectionList::class)]12final class SelectionListTest extends TestCase13final class SelectionListTest extends TestCase13{14{14-public function testGetEntityIds(): void15+private const int CHUNK_SIZE = 100;16+17+public function testAddEntityListAndGetAllIds(): void18+ {19+$selectionList = new SelectionList();20+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);21+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);22+23+$this->assertSame([1, 2, 3, 4], $selectionList->getAllIds());24+ }25+26+public function testAddEntityListMergesMultipleCallsForSamePriority(): void27+ {28+$selectionList = new SelectionList();29+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);30+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [3, 4]);31+32+$this->assertSame([1, 2, 3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));33+ }34+35+public function testGetEntityListItemsByGroup(): void36+ {37+$selectionList = new SelectionList();38+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);39+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);40+$selectionList->addEntityList(QueuePriorityEnum::LOW, [5, 6]);41+42+$this->assertSame([1, 2], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));43+$this->assertSame([3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::NORMAL));44+$this->assertSame([5, 6], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::LOW));45+ }46+47+public function testGetEntityListItemsByGroupReturnsEmptyArrayForUnknownGroup(): void15 {48 {16-$priority = ['p1', 'p2'];49+$selectionList = new SelectionList();17-$normal = ['n1', 'n2'];50+18-$selectionList = new SelectionList($priority, $normal);51+$this->assertSame([], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));52+ }195320-$this->assertSame($priority, $selectionList->getHighPriorityEntityIds());54+public function testGetEntityListGroups(): void21-$this->assertSame($normal, $selectionList->getNormalPriorityEntityId());55+ {56+$selectionList = new SelectionList();57+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);58+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2]);225923-$this->assertSame(['p1', 'p2', 'n1', 'n2'], $selectionList->getAllIds());60+$this->assertSame(['high', 'normal'], $selectionList->getEntityListGroups());24 }61 }256226public function testCountReturnsTotalCount(): void63public function testCountReturnsTotalCount(): void27 {64 {28-$selectionList = new SelectionList(['p1'], ['n1', 'n2']);65+$selectionList = new SelectionList();66+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);67+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2, 3]);68+29$this->assertSame(3, $selectionList->count());69$this->assertSame(3, $selectionList->count());30 }70 }317132-public function testIsEmptyReturnsTrueWhenBothListsAreEmpty(): void72+public function testIsEmptyReturnsTrueWhenNoEntitiesAdded(): void33 {73 {34-$selectionList = new SelectionList([], []);74+$selectionList = new SelectionList();75+35$this->assertTrue($selectionList->isEmpty());76$this->assertTrue($selectionList->isEmpty());36 }77 }377838-public function testIsEmptyReturnsFalseWhenAnyListIsNotEmpty(): void79+public function testIsEmptyReturnsFalseWhenEntitiesAdded(): void39 {80 {40-$selectionList = new SelectionList(['p1'], []);81+$selectionList = new SelectionList();41-$this->assertFalse($selectionList->isEmpty());82+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);428343-$selectionList2 = new SelectionList([], ['n1']);84+$this->assertFalse($selectionList->isEmpty());44-$this->assertFalse($selectionList2->isEmpty());45 }85 }468647-public function testIsStaleChunkSelected(): void87+public function testMarkStaleAndIsStaleChunkSelected(): void48 {88 {49-$selectionList = new SelectionList([], [], true);89+$selectionList = new SelectionList();50-$this->assertTrue($selectionList->isStaleChunkSelected());90+91+$this->assertFalse($selectionList->isStaleChunkSelected());92+93+$selectionList->markStale();519452-$selectionList2 = new SelectionList([], [], false);95+$this->assertTrue($selectionList->isStaleChunkSelected());53-$this->assertFalse($selectionList2->isStaleChunkSelected());54 }96 }559756public function testIsFullChunk(): void98public function testIsFullChunk(): void57 {99 {58-$chunkSize = 200;100+$selectionList = new SelectionList();59-101+$selectionList->setChunkSize(self::CHUNK_SIZE);60-$fullList = range(1, $chunkSize);102+61-$selectionList = new SelectionList(103+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, self::CHUNK_SIZE));62- priorityEntityIdList: $fullList,104+63- normalEntityIdList: [],105+$this->assertTrue($selectionList->isFullChunk());64- chunkSize: $chunkSize,106+ }65- );107+108+public function testIsNotFullChunkWhenBelowChunkSize(): void109+ {110+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);111+112+$selectionList = new SelectionList();113+$selectionList->setChunkSize(self::CHUNK_SIZE);114+115+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range(1, $halfRange));116+117+$this->assertFalse($selectionList->isFullChunk());118+ }119+120+public function testIsFullChunkAcrossMultiplePriorityGroups(): void121+ {122+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);123+124+$selectionList = new SelectionList();125+$selectionList->setChunkSize(self::CHUNK_SIZE);126+127+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, $halfRange));128+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range($halfRange + 1, self::CHUNK_SIZE));129+130+$this->assertTrue($selectionList->isFullChunk());131+ }132+133+public function testSetChunkSizeAffectsIsFullChunk(): void134+ {135+$selectionList = new SelectionList();136+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [1, 2, 3]);137+138+$selectionList->setChunkSize(3);66$this->assertTrue($selectionList->isFullChunk());139$this->assertTrue($selectionList->isFullChunk());6714068-$halfRange = (int) ceil($chunkSize / 2);141+$selectionList->setChunkSize(50);69-142+$this->assertFalse($selectionList->isFullChunk());70-$notFullList = range(1, $halfRange);71-$selectionList2 = new SelectionList(72- priorityEntityIdList: $notFullList,73- normalEntityIdList: [],74- chunkSize: $chunkSize,75- );76-$this->assertFalse($selectionList2->isFullChunk());77-78-$priorityRange = range(1, $halfRange);79-$normalRange = range($halfRange + 1, $chunkSize);80-81-$selectionList3 = new SelectionList(82- priorityEntityIdList: $priorityRange,83- normalEntityIdList: $normalRange,84- chunkSize: $chunkSize,85- );86-$this->assertTrue($selectionList3->isFullChunk());87 }143 }88}144}</selection>” selected. Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
You said
I’m on page “<tabTitle>JY-20808 low priority indexing queue by Vasil-Jimi</tabTitle>” with “<selection>@@ -5,84 +5,140 @@5namespace Tests\Unit\Component\ES\Processor\DTOs;5namespace Tests\Unit\Component\ES\Processor\DTOs;667use Jiminny\Component\ES\Processor\DTOs\SelectionList;7use Jiminny\Component\ES\Processor\DTOs\SelectionList;8+use Jiminny\Component\ES\QueuePriorityEnum;8use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\TestCase;10use PHPUnit\Framework\TestCase;101111#[CoversClass(SelectionList::class)]12#[CoversClass(SelectionList::class)]12final class SelectionListTest extends TestCase13final class SelectionListTest extends TestCase13{14{14-public function testGetEntityIds(): void15+private const int CHUNK_SIZE = 100;16+17+public function testAddEntityListAndGetAllIds(): void18+ {19+$selectionList = new SelectionList();20+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);21+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);22+23+$this->assertSame([1, 2, 3, 4], $selectionList->getAllIds());24+ }25+26+public function testAddEntityListMergesMultipleCallsForSamePriority(): void27+ {28+$selectionList = new SelectionList();29+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);30+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [3, 4]);31+32+$this->assertSame([1, 2, 3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));33+ }34+35+public function testGetEntityListItemsByGroup(): void36+ {37+$selectionList = new SelectionList();38+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);39+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);40+$selectionList->addEntityList(QueuePriorityEnum::LOW, [5, 6]);41+42+$this->assertSame([1, 2], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));43+$this->assertSame([3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::NORMAL));44+$this->assertSame([5, 6], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::LOW));45+ }46+47+public function testGetEntityListItemsByGroupReturnsEmptyArrayForUnknownGroup(): void15 {48 {16-$priority = ['p1', 'p2'];49+$selectionList = new SelectionList();17-$normal = ['n1', 'n2'];50+18-$selectionList = new SelectionList($priority, $normal);51+$this->assertSame([], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));52+ }195320-$this->assertSame($priority, $selectionList->getHighPriorityEntityIds());54+public function testGetEntityListGroups(): void21-$this->assertSame($normal, $selectionList->getNormalPriorityEntityId());55+ {56+$selectionList = new SelectionList();57+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);58+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2]);225923-$this->assertSame(['p1', 'p2', 'n1', 'n2'], $selectionList->getAllIds());60+$this->assertSame(['high', 'normal'], $selectionList->getEntityListGroups());24 }61 }256226public function testCountReturnsTotalCount(): void63public function testCountReturnsTotalCount(): void27 {64 {28-$selectionList = new SelectionList(['p1'], ['n1', 'n2']);65+$selectionList = new SelectionList();66+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);67+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2, 3]);68+29$this->assertSame(3, $selectionList->count());69$this->assertSame(3, $selectionList->count());30 }70 }317132-public function testIsEmptyReturnsTrueWhenBothListsAreEmpty(): void72+public function testIsEmptyReturnsTrueWhenNoEntitiesAdded(): void33 {73 {34-$selectionList = new SelectionList([], []);74+$selectionList = new SelectionList();75+35$this->assertTrue($selectionList->isEmpty());76$this->assertTrue($selectionList->isEmpty());36 }77 }377838-public function testIsEmptyReturnsFalseWhenAnyListIsNotEmpty(): void79+public function testIsEmptyReturnsFalseWhenEntitiesAdded(): void39 {80 {40-$selectionList = new SelectionList(['p1'], []);81+$selectionList = new SelectionList();41-$this->assertFalse($selectionList->isEmpty());82+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);428343-$selectionList2 = new SelectionList([], ['n1']);84+$this->assertFalse($selectionList->isEmpty());44-$this->assertFalse($selectionList2->isEmpty());45 }85 }468647-public function testIsStaleChunkSelected(): void87+public function testMarkStaleAndIsStaleChunkSelected(): void48 {88 {49-$selectionList = new SelectionList([], [], true);89+$selectionList = new SelectionList();50-$this->assertTrue($selectionList->isStaleChunkSelected());90+91+$this->assertFalse($selectionList->isStaleChunkSelected());92+93+$selectionList->markStale();519452-$selectionList2 = new SelectionList([], [], false);95+$this->assertTrue($selectionList->isStaleChunkSelected());53-$this->assertFalse($selectionList2->isStaleChunkSelected());54 }96 }559756public function testIsFullChunk(): void98public function testIsFullChunk(): void57 {99 {58-$chunkSize = 200;100+$selectionList = new SelectionList();59-101+$selectionList->setChunkSize(self::CHUNK_SIZE);60-$fullList = range(1, $chunkSize);102+61-$selectionList = new SelectionList(103+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, self::CHUNK_SIZE));62- priorityEntityIdList: $fullList,104+63- normalEntityIdList: [],105+$this->assertTrue($selectionList->isFullChunk());64- chunkSize: $chunkSize,106+ }65- );107+108+public function testIsNotFullChunkWhenBelowChunkSize(): void109+ {110+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);111+112+$selectionList = new SelectionList();113+$selectionList->setChunkSize(self::CHUNK_SIZE);114+115+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range(1, $halfRange));116+117+$this->assertFalse($selectionList->isFullChunk());118+ }119+120+public function testIsFullChunkAcrossMultiplePriorityGroups(): void121+ {122+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);123+124+$selectionList = new SelectionList();125+$selectionList->setChunkSize(self::CHUNK_SIZE);126+127+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, $halfRange));128+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range($halfRange + 1, self::CHUNK_SIZE));129+130+$this->assertTrue($selectionList->isFullChunk());131+ }132+133+public function testSetChunkSizeAffectsIsFullChunk(): void134+ {135+$selectionList = new SelectionList();136+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [1, 2, 3]);137+138+$selectionList->setChunkSize(3);66$this->assertTrue($selectionList->isFullChunk());139$this->assertTrue($selectionList->isFullChunk());6714068-$halfRange = (int) ceil($chunkSize / 2);141+$selectionList->setChunkSize(50);69-142+$this->assertFalse($selectionList->isFullChunk());70-$notFullList = range(1, $halfRange);71-$selectionList2 = new SelectionList(72- priorityEntityIdList: $notFullList,73- normalEntityIdList: [],74- chunkSize: $chunkSize,75- );76-$this->assertFalse($selectionList2->isFullChunk());77-78-$priorityRange = range(1, $halfRange);79-$normalRange = range($halfRange + 1, $chunkSize);80-81-$selectionList3 = new SelectionList(82- priorityEntityIdList: $priorityRange,83- normalEntityIdList: $normalRange,84- chunkSize: $chunkSize,85- );86-$this->assertTrue($selectionList3->isFullChunk());87 }143 }88}144}</selection>” selected.
Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
Expand
Listen
Gemini said
Gemini said
Here is a precise and concise summary of the code selection, which refactors
SelectionListTest
to accommodate a shift from fixed high/normal priority properties to a more dynamic, enum-driven priority group architecture.
Refactored Priority Handling
Refactored Priority Handling
Dynamic Priority Groups:
Replaces fixed high/normal array parameters with the
QueuePriorityEnum
(
HIGH
,
NORMAL
,
LOW
), allowing entity lists to be managed dynamically by group types.
Merged Registrations:
Validates that multiple
addEntityList()
calls targeting the same priority level properly merge their data instead of overwriting it.
Group Querying:
Tests new methods
getEntityListItemsByGroup()
and
getEntityListGroups()
to retrieve group-specific IDs or an array of active group names (e.g.,
['high', 'normal']
).
Updated State and Count Validations
Updated State and Count Validations
Instantiation:
Tests now instantiate
SelectionList
without initial constructor arguments, using setters and action methods instead.
Empty States:
Updates assertions (
isEmpty()
) to verify state based on whether any entities have been dynamically added, rather than checking pre-defined constructor arrays.
Stale Flags:
Changes from passing a
stale
boolean via the constructor to explicitly triggering a mutable
$selectionList->markStale()
method.
Chunk Capacity Behavior
Chunk Capacity Behavior
Flexible Chunk Sizes:
Introduces a
setChunkSize()
method, replacing the constructor-configured chunk boundary.
Capacity Assertions:
Retains and updates the logic for
isFullChunk()
, ensuring it accurately evaluates capacity when elements span multiple dynamic priority groups or when the chunk limit is altered mid-lifecycle.
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said Are there any issues or potential bolltenecks
You said
Are there any issues or potential bolltenecks
Listen
Show thinking
Gemini said
Gemini said
Based on the test diff you provided, we can infer a few potential issues and architectural bottlenecks in the underlying
SelectionList
implementation. While the tests are well-structured, the behavior they describe hints at a few areas you might want to keep an eye on, particularly for an Elasticsearch (ES) processor queue.
Here are the potential issues and bottlenecks to consider:
1. Array Merging Overhead (Performance)
1. Array Merging Overhead (Performance)
The
testAddEntityListMergesMultipleCallsForSamePriority
test shows that calling
addEntityList()
multiple times for the same priority group merges the arrays.
The Bottleneck:
If the underlying implementation uses
array_merge()
inside a loop to append these IDs, it creates a new array in memory every time it's called. If you are processing thousands of entities and appending them in small batches, this can lead to memory bloat and CPU degradation due to constant array reallocation.
Mitigation:
Ensure the implementation handles batch appends efficiently (e.g., using the spread operator
[...$existing, ...$new]
or appending via references) or groups the arrays before doing a single merge.
2. Lack of Uniqueness/Deduplication (Logic)
2. Lack of Uniqueness/Deduplication (Logic)
None of the tests explicitly verify what happens if duplicate IDs are added. For example, if you call
addEntityList(QueuePriorityEnum::HIGH, [1, 2])
and then
addEntityList(QueuePriorityEnum::HIGH, [2, 3])
.
The Issue:
Does the getter return
[1, 2, 2, 3]
or
[1, 2, 3]
? If duplicates are not filtered out, your ES processor might end up indexing the same entity multiple times in a single chunk, which wastes I/O and processor cycles.
Mitigation:
The underlying class should ideally enforce uniqueness (e.g., using
array_unique()
before resolving the chunk, or using ID keys as associative array keys to naturally prevent duplicates).
3. Memory Consumption with Large Chunks (Scalability)
3. Memory Consumption with Large Chunks (Scalability)
The tests introduce
setChunkSize()
. If this chunk size is configured to be very large (e.g., tens of thousands of IDs to bulk-send to Elasticsearch).
The Bottleneck:
Holding massive arrays of IDs in memory across different priority groups can quickly hit PHP memory limits, especially if this
SelectionList
object is kept alive during a long-running daemon or cron job.
Mitigation:
Monitor memory usage if
CHUNK_SIZE
grows, and ensure the
SelectionList
object is properly destroyed (or garbage collected) after the chunk is processed to free up memory.
4. Type Juggling
4. Type Juggling
In the original code, string IDs were used (
['p1', 'p2']
). In the refactored tests, integer IDs are used (
[1, 2]
).
The Issue:
If your system handles UUIDs (strings) vs. relational DB auto-increments (integers), make sure the underlying implementation uses strict comparisons (
===
) where necessary, especially if deduplicating arrays, to prevent PHP from juggling
'1'
and
1
improperly.
Would you like me to draft a quick test case to check how the class currently handles duplicate IDs, or is deduplication intentionally handled further down your pipeline?
Good response...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes
|
55647
|
|
55648
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Copy prompt
You said I’m on page “<tabTitle>JY-20808 low priority indexing queue by Vasil-Jimi</tabTitle>” with “<selection>@@ -5,84 +5,140 @@5namespace Tests\Unit\Component\ES\Processor\DTOs;5namespace Tests\Unit\Component\ES\Processor\DTOs;667use Jiminny\Component\ES\Processor\DTOs\SelectionList;7use Jiminny\Component\ES\Processor\DTOs\SelectionList;8+use Jiminny\Component\ES\QueuePriorityEnum;8use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\TestCase;10use PHPUnit\Framework\TestCase;101111#[CoversClass(SelectionList::class)]12#[CoversClass(SelectionList::class)]12final class SelectionListTest extends TestCase13final class SelectionListTest extends TestCase13{14{14-public function testGetEntityIds(): void15+private const int CHUNK_SIZE = 100;16+17+public function testAddEntityListAndGetAllIds(): void18+ {19+$selectionList = new SelectionList();20+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);21+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);22+23+$this->assertSame([1, 2, 3, 4], $selectionList->getAllIds());24+ }25+26+public function testAddEntityListMergesMultipleCallsForSamePriority(): void27+ {28+$selectionList = new SelectionList();29+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);30+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [3, 4]);31+32+$this->assertSame([1, 2, 3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));33+ }34+35+public function testGetEntityListItemsByGroup(): void36+ {37+$selectionList = new SelectionList();38+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);39+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);40+$selectionList->addEntityList(QueuePriorityEnum::LOW, [5, 6]);41+42+$this->assertSame([1, 2], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));43+$this->assertSame([3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::NORMAL));44+$this->assertSame([5, 6], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::LOW));45+ }46+47+public function testGetEntityListItemsByGroupReturnsEmptyArrayForUnknownGroup(): void15 {48 {16-$priority = ['p1', 'p2'];49+$selectionList = new SelectionList();17-$normal = ['n1', 'n2'];50+18-$selectionList = new SelectionList($priority, $normal);51+$this->assertSame([], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));52+ }195320-$this->assertSame($priority, $selectionList->getHighPriorityEntityIds());54+public function testGetEntityListGroups(): void21-$this->assertSame($normal, $selectionList->getNormalPriorityEntityId());55+ {56+$selectionList = new SelectionList();57+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);58+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2]);225923-$this->assertSame(['p1', 'p2', 'n1', 'n2'], $selectionList->getAllIds());60+$this->assertSame(['high', 'normal'], $selectionList->getEntityListGroups());24 }61 }256226public function testCountReturnsTotalCount(): void63public function testCountReturnsTotalCount(): void27 {64 {28-$selectionList = new SelectionList(['p1'], ['n1', 'n2']);65+$selectionList = new SelectionList();66+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);67+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2, 3]);68+29$this->assertSame(3, $selectionList->count());69$this->assertSame(3, $selectionList->count());30 }70 }317132-public function testIsEmptyReturnsTrueWhenBothListsAreEmpty(): void72+public function testIsEmptyReturnsTrueWhenNoEntitiesAdded(): void33 {73 {34-$selectionList = new SelectionList([], []);74+$selectionList = new SelectionList();75+35$this->assertTrue($selectionList->isEmpty());76$this->assertTrue($selectionList->isEmpty());36 }77 }377838-public function testIsEmptyReturnsFalseWhenAnyListIsNotEmpty(): void79+public function testIsEmptyReturnsFalseWhenEntitiesAdded(): void39 {80 {40-$selectionList = new SelectionList(['p1'], []);81+$selectionList = new SelectionList();41-$this->assertFalse($selectionList->isEmpty());82+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);428343-$selectionList2 = new SelectionList([], ['n1']);84+$this->assertFalse($selectionList->isEmpty());44-$this->assertFalse($selectionList2->isEmpty());45 }85 }468647-public function testIsStaleChunkSelected(): void87+public function testMarkStaleAndIsStaleChunkSelected(): void48 {88 {49-$selectionList = new SelectionList([], [], true);89+$selectionList = new SelectionList();50-$this->assertTrue($selectionList->isStaleChunkSelected());90+91+$this->assertFalse($selectionList->isStaleChunkSelected());92+93+$selectionList->markStale();519452-$selectionList2 = new SelectionList([], [], false);95+$this->assertTrue($selectionList->isStaleChunkSelected());53-$this->assertFalse($selectionList2->isStaleChunkSelected());54 }96 }559756public function testIsFullChunk(): void98public function testIsFullChunk(): void57 {99 {58-$chunkSize = 200;100+$selectionList = new SelectionList();59-101+$selectionList->setChunkSize(self::CHUNK_SIZE);60-$fullList = range(1, $chunkSize);102+61-$selectionList = new SelectionList(103+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, self::CHUNK_SIZE));62- priorityEntityIdList: $fullList,104+63- normalEntityIdList: [],105+$this->assertTrue($selectionList->isFullChunk());64- chunkSize: $chunkSize,106+ }65- );107+108+public function testIsNotFullChunkWhenBelowChunkSize(): void109+ {110+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);111+112+$selectionList = new SelectionList();113+$selectionList->setChunkSize(self::CHUNK_SIZE);114+115+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range(1, $halfRange));116+117+$this->assertFalse($selectionList->isFullChunk());118+ }119+120+public function testIsFullChunkAcrossMultiplePriorityGroups(): void121+ {122+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);123+124+$selectionList = new SelectionList();125+$selectionList->setChunkSize(self::CHUNK_SIZE);126+127+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, $halfRange));128+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range($halfRange + 1, self::CHUNK_SIZE));129+130+$this->assertTrue($selectionList->isFullChunk());131+ }132+133+public function testSetChunkSizeAffectsIsFullChunk(): void134+ {135+$selectionList = new SelectionList();136+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [1, 2, 3]);137+138+$selectionList->setChunkSize(3);66$this->assertTrue($selectionList->isFullChunk());139$this->assertTrue($selectionList->isFullChunk());6714068-$halfRange = (int) ceil($chunkSize / 2);141+$selectionList->setChunkSize(50);69-142+$this->assertFalse($selectionList->isFullChunk());70-$notFullList = range(1, $halfRange);71-$selectionList2 = new SelectionList(72- priorityEntityIdList: $notFullList,73- normalEntityIdList: [],74- chunkSize: $chunkSize,75- );76-$this->assertFalse($selectionList2->isFullChunk());77-78-$priorityRange = range(1, $halfRange);79-$normalRange = range($halfRange + 1, $chunkSize);80-81-$selectionList3 = new SelectionList(82- priorityEntityIdList: $priorityRange,83- normalEntityIdList: $normalRange,84- chunkSize: $chunkSize,85- );86-$this->assertTrue($selectionList3->isFullChunk());87 }143 }88}144}</selection>” selected. Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
You said
I’m on page “<tabTitle>JY-20808 low priority indexing queue by Vasil-Jimi</tabTitle>” with “<selection>@@ -5,84 +5,140 @@5namespace Tests\Unit\Component\ES\Processor\DTOs;5namespace Tests\Unit\Component\ES\Processor\DTOs;667use Jiminny\Component\ES\Processor\DTOs\SelectionList;7use Jiminny\Component\ES\Processor\DTOs\SelectionList;8+use Jiminny\Component\ES\QueuePriorityEnum;8use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\TestCase;10use PHPUnit\Framework\TestCase;101111#[CoversClass(SelectionList::class)]12#[CoversClass(SelectionList::class)]12final class SelectionListTest extends TestCase13final class SelectionListTest extends TestCase13{14{14-public function testGetEntityIds(): void15+private const int CHUNK_SIZE = 100;16+17+public function testAddEntityListAndGetAllIds(): void18+ {19+$selectionList = new SelectionList();20+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);21+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);22+23+$this->assertSame([1, 2, 3, 4], $selectionList->getAllIds());24+ }25+26+public function testAddEntityListMergesMultipleCallsForSamePriority(): void27+ {28+$selectionList = new SelectionList();29+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);30+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [3, 4]);31+32+$this->assertSame([1, 2, 3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));33+ }34+35+public function testGetEntityListItemsByGroup(): void36+ {37+$selectionList = new SelectionList();38+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);39+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);40+$selectionList->addEntityList(QueuePriorityEnum::LOW, [5, 6]);41+42+$this->assertSame([1, 2], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));43+$this->assertSame([3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::NORMAL));44+$this->assertSame([5, 6], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::LOW));45+ }46+47+public function testGetEntityListItemsByGroupReturnsEmptyArrayForUnknownGroup(): void15 {48 {16-$priority = ['p1', 'p2'];49+$selectionList = new SelectionList();17-$normal = ['n1', 'n2'];50+18-$selectionList = new SelectionList($priority, $normal);51+$this->assertSame([], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));52+ }195320-$this->assertSame($priority, $selectionList->getHighPriorityEntityIds());54+public function testGetEntityListGroups(): void21-$this->assertSame($normal, $selectionList->getNormalPriorityEntityId());55+ {56+$selectionList = new SelectionList();57+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);58+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2]);225923-$this->assertSame(['p1', 'p2', 'n1', 'n2'], $selectionList->getAllIds());60+$this->assertSame(['high', 'normal'], $selectionList->getEntityListGroups());24 }61 }256226public function testCountReturnsTotalCount(): void63public function testCountReturnsTotalCount(): void27 {64 {28-$selectionList = new SelectionList(['p1'], ['n1', 'n2']);65+$selectionList = new SelectionList();66+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);67+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2, 3]);68+29$this->assertSame(3, $selectionList->count());69$this->assertSame(3, $selectionList->count());30 }70 }317132-public function testIsEmptyReturnsTrueWhenBothListsAreEmpty(): void72+public function testIsEmptyReturnsTrueWhenNoEntitiesAdded(): void33 {73 {34-$selectionList = new SelectionList([], []);74+$selectionList = new SelectionList();75+35$this->assertTrue($selectionList->isEmpty());76$this->assertTrue($selectionList->isEmpty());36 }77 }377838-public function testIsEmptyReturnsFalseWhenAnyListIsNotEmpty(): void79+public function testIsEmptyReturnsFalseWhenEntitiesAdded(): void39 {80 {40-$selectionList = new SelectionList(['p1'], []);81+$selectionList = new SelectionList();41-$this->assertFalse($selectionList->isEmpty());82+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);428343-$selectionList2 = new SelectionList([], ['n1']);84+$this->assertFalse($selectionList->isEmpty());44-$this->assertFalse($selectionList2->isEmpty());45 }85 }468647-public function testIsStaleChunkSelected(): void87+public function testMarkStaleAndIsStaleChunkSelected(): void48 {88 {49-$selectionList = new SelectionList([], [], true);89+$selectionList = new SelectionList();50-$this->assertTrue($selectionList->isStaleChunkSelected());90+91+$this->assertFalse($selectionList->isStaleChunkSelected());92+93+$selectionList->markStale();519452-$selectionList2 = new SelectionList([], [], false);95+$this->assertTrue($selectionList->isStaleChunkSelected());53-$this->assertFalse($selectionList2->isStaleChunkSelected());54 }96 }559756public function testIsFullChunk(): void98public function testIsFullChunk(): void57 {99 {58-$chunkSize = 200;100+$selectionList = new SelectionList();59-101+$selectionList->setChunkSize(self::CHUNK_SIZE);60-$fullList = range(1, $chunkSize);102+61-$selectionList = new SelectionList(103+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, self::CHUNK_SIZE));62- priorityEntityIdList: $fullList,104+63- normalEntityIdList: [],105+$this->assertTrue($selectionList->isFullChunk());64- chunkSize: $chunkSize,106+ }65- );107+108+public function testIsNotFullChunkWhenBelowChunkSize(): void109+ {110+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);111+112+$selectionList = new SelectionList();113+$selectionList->setChunkSize(self::CHUNK_SIZE);114+115+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range(1, $halfRange));116+117+$this->assertFalse($selectionList->isFullChunk());118+ }119+120+public function testIsFullChunkAcrossMultiplePriorityGroups(): void121+ {122+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);123+124+$selectionList = new SelectionList();125+$selectionList->setChunkSize(self::CHUNK_SIZE);126+127+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, $halfRange));128+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range($halfRange + 1, self::CHUNK_SIZE));129+130+$this->assertTrue($selectionList->isFullChunk());131+ }132+133+public function testSetChunkSizeAffectsIsFullChunk(): void134+ {135+$selectionList = new SelectionList();136+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [1, 2, 3]);137+138+$selectionList->setChunkSize(3);66$this->assertTrue($selectionList->isFullChunk());139$this->assertTrue($selectionList->isFullChunk());6714068-$halfRange = (int) ceil($chunkSize / 2);141+$selectionList->setChunkSize(50);69-142+$this->assertFalse($selectionList->isFullChunk());70-$notFullList = range(1, $halfRange);71-$selectionList2 = new SelectionList(72- priorityEntityIdList: $notFullList,73- normalEntityIdList: [],74- chunkSize: $chunkSize,75- );76-$this->assertFalse($selectionList2->isFullChunk());77-78-$priorityRange = range(1, $halfRange);79-$normalRange = range($halfRange + 1, $chunkSize);80-81-$selectionList3 = new SelectionList(82- priorityEntityIdList: $priorityRange,83- normalEntityIdList: $normalRange,84- chunkSize: $chunkSize,85- );86-$this->assertTrue($selectionList3->isFullChunk());87 }143 }88}144}</selection>” selected.
Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
Expand
Listen
Gemini said
Gemini said
Here is a precise and concise summary of the code selection, which refactors
SelectionListTest
to accommodate a shift from fixed high/normal priority properties to a more dynamic, enum-driven priority group architecture.
Refactored Priority Handling
Refactored Priority Handling
Dynamic Priority Groups:
Replaces fixed high/normal array parameters with the
QueuePriorityEnum
(
HIGH
,
NORMAL
,
LOW
), allowing entity lists to be managed dynamically by group types.
Merged Registrations:
Validates that multiple
addEntityList()
calls targeting the same priority level properly merge their data instead of overwriting it.
Group Querying:
Tests new methods
getEntityListItemsByGroup()
and
getEntityListGroups()
to retrieve group-specific IDs or an array of active group names (e.g.,
['high', 'normal']
).
Updated State and Count Validations
Updated State and Count Validations
Instantiation:
Tests now instantiate
SelectionList
without initial constructor arguments, using setters and action methods instead.
Empty States:
Updates assertions (
isEmpty()
) to verify state based on whether any entities have been dynamically added, rather than checking pre-defined constructor arrays.
Stale Flags:
Changes from passing a
stale
boolean via the constructor to explicitly triggering a mutable
$selectionList->markStale()
method.
Chunk Capacity Behavior
Chunk Capacity Behavior
Flexible Chunk Sizes:
Introduces a
setChunkSize()
method, replacing the constructor-configured chunk boundary.
Capacity Assertions:
Retains and updates the logic for
isFullChunk()
, ensuring it accurately evaluates capacity when elements span multiple dynamic priority groups or when the chunk limit is altered mid-lifecycle.
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
You said Are there any issues or potential bolltenecks
You said
Are there any issues or potential bolltenecks
Listen
Show thinking
Gemini said
Gemini said
Based on the test diff you provided, we can infer a few potential issues and architectural bottlenecks in the underlying
SelectionList
implementation. While the tests are well-structured, the behavior they describe hints at a few areas you might want to keep an eye on, particularly for an Elasticsearch (ES) processor queue.
Here are the potential issues and bottlenecks to consider:
1. Array Merging Overhead (Performance)
1. Array Merging Overhead (Performance)
The
testAddEntityListMergesMultipleCallsForSamePriority
test shows that calling
addEntityList()
multiple times for the same priority group merges the arrays.
The Bottleneck:
If the underlying implementation uses
array_merge()
inside a loop to append these IDs, it creates a new array in memory every time it's called. If you are processing thousands of entities and appending them in small batches, this can lead to memory bloat and CPU degradation due to constant array reallocation.
Mitigation:
Ensure the implementation handles batch appends efficiently (e.g., using the spread operator
[...$existing, ...$new]
or appending via references) or groups the arrays before doing a single merge.
2. Lack of Uniqueness/Deduplication (Logic)
2. Lack of Uniqueness/Deduplication (Logic)
None of the tests explicitly verify what happens if duplicate IDs are added. For example, if you call
addEntityList(QueuePriorityEnum::HIGH, [1, 2])
and then
addEntityList(QueuePriorityEnum::HIGH, [2, 3])
.
The Issue:
Does the getter return
[1, 2, 2, 3]
or
[1, 2, 3]
? If duplicates are not filtered out, your ES processor might end up indexing the same entity multiple times in a single chunk, which wastes I/O and processor cycles.
Mitigation:
The underlying class should ideally enforce uniqueness (e.g., using
array_unique()
before resolving the chunk, or using ID keys as associative array keys to naturally prevent duplicates).
3. Memory Consumption with Large Chunks (Scalability)
3. Memory Consumption with Large Chunks (Scalability)
The tests introduce
setChunkSize()
. If this chunk size is configured to be very large (e.g., tens of thousands of IDs to bulk-send to Elasticsearch).
The Bottleneck:
Holding massive arrays of IDs in memory across different priority groups can quickly hit PHP memory limits, especially if this
SelectionList
object is kept alive during a long-running daemon or cron job.
Mitigation:
Monitor memory usage if
CHUNK_SIZE
grows, and ensure the
SelectionList
object is properly destroyed (or garbage collected) after the chunk is processed to free up memory.
4. Type Juggling
4. Type Juggling
In the original code, string IDs were used (
['p1', 'p2']
). In the refactored tests, integer IDs are used (
[1, 2]
).
The Issue:
If your system handles UUIDs (strings) vs. relational DB auto-increments (integers), make sure the underlying implementation uses strict comparisons (
===
) where necessary, especially if deduplicating arrays, to prevent PHP from juggling
'1'
and
1
improperly.
Would you like me to draft a quick test case to check how the class currently handles duplicate IDs, or is deduplication intentionally handled further down your pipeline?
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said what about non test files
You said
what about non test files
Enter a prompt for Gemini
encrypted
Enter a prompt for Gemini
encrypted
Open upload file menu
Tools
Open mode picker
Pro
Stop response
Your Jiminny chats aren’t used to improve our models. Gemini is AI. It can make mistakes, so double check it.
Your privacy & Gemini Opens in a new window
Your privacy & Gemini
Opens in a new window
Gemini is typing
Summarize page
Summarize page
Skip to content
Skip to content
Open menu
Homepage (g then d)
jiminny
jiminny
app
app
Search or jump to…
Type
/
to search
Chat with Copilot
Open Copilot…
Create new...
All issues(g then i)
All pull requests
All repositories
You have unread notifications(g then n)
Open user navigation menu
Repository navigation
Repository navigation
Code
Code
Pull requests (31)
Pull requests
(
31
)
Agents
Agents
Actions
Actions
Wiki
Wiki
Security and quality (1)
Security and quality
(
1
)
Insights
Insights
Settings
Settings
Important update
Important update
On April 24 we'll start using GitHub Copilot interaction data for AI model training unless you opt out.
Review this update
Review this update
and manage your preferences in your
GitHub account settings
GitHub account settings
.
Dismiss banner...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes
|
55648
|
|
55649
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New chat
Gemini
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Copy prompt
You said I’m on page “<tabTitle>JY-20808 low priority indexing queue by Vasil-Jimi</tabTitle>” with “<selection>@@ -5,84 +5,140 @@5namespace Tests\Unit\Component\ES\Processor\DTOs;5namespace Tests\Unit\Component\ES\Processor\DTOs;667use Jiminny\Component\ES\Processor\DTOs\SelectionList;7use Jiminny\Component\ES\Processor\DTOs\SelectionList;8+use Jiminny\Component\ES\QueuePriorityEnum;8use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\TestCase;10use PHPUnit\Framework\TestCase;101111#[CoversClass(SelectionList::class)]12#[CoversClass(SelectionList::class)]12final class SelectionListTest extends TestCase13final class SelectionListTest extends TestCase13{14{14-public function testGetEntityIds(): void15+private const int CHUNK_SIZE = 100;16+17+public function testAddEntityListAndGetAllIds(): void18+ {19+$selectionList = new SelectionList();20+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);21+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);22+23+$this->assertSame([1, 2, 3, 4], $selectionList->getAllIds());24+ }25+26+public function testAddEntityListMergesMultipleCallsForSamePriority(): void27+ {28+$selectionList = new SelectionList();29+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);30+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [3, 4]);31+32+$this->assertSame([1, 2, 3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));33+ }34+35+public function testGetEntityListItemsByGroup(): void36+ {37+$selectionList = new SelectionList();38+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);39+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);40+$selectionList->addEntityList(QueuePriorityEnum::LOW, [5, 6]);41+42+$this->assertSame([1, 2], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));43+$this->assertSame([3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::NORMAL));44+$this->assertSame([5, 6], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::LOW));45+ }46+47+public function testGetEntityListItemsByGroupReturnsEmptyArrayForUnknownGroup(): void15 {48 {16-$priority = ['p1', 'p2'];49+$selectionList = new SelectionList();17-$normal = ['n1', 'n2'];50+18-$selectionList = new SelectionList($priority, $normal);51+$this->assertSame([], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));52+ }195320-$this->assertSame($priority, $selectionList->getHighPriorityEntityIds());54+public function testGetEntityListGroups(): void21-$this->assertSame($normal, $selectionList->getNormalPriorityEntityId());55+ {56+$selectionList = new SelectionList();57+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);58+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2]);225923-$this->assertSame(['p1', 'p2', 'n1', 'n2'], $selectionList->getAllIds());60+$this->assertSame(['high', 'normal'], $selectionList->getEntityListGroups());24 }61 }256226public function testCountReturnsTotalCount(): void63public function testCountReturnsTotalCount(): void27 {64 {28-$selectionList = new SelectionList(['p1'], ['n1', 'n2']);65+$selectionList = new SelectionList();66+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);67+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2, 3]);68+29$this->assertSame(3, $selectionList->count());69$this->assertSame(3, $selectionList->count());30 }70 }317132-public function testIsEmptyReturnsTrueWhenBothListsAreEmpty(): void72+public function testIsEmptyReturnsTrueWhenNoEntitiesAdded(): void33 {73 {34-$selectionList = new SelectionList([], []);74+$selectionList = new SelectionList();75+35$this->assertTrue($selectionList->isEmpty());76$this->assertTrue($selectionList->isEmpty());36 }77 }377838-public function testIsEmptyReturnsFalseWhenAnyListIsNotEmpty(): void79+public function testIsEmptyReturnsFalseWhenEntitiesAdded(): void39 {80 {40-$selectionList = new SelectionList(['p1'], []);81+$selectionList = new SelectionList();41-$this->assertFalse($selectionList->isEmpty());82+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);428343-$selectionList2 = new SelectionList([], ['n1']);84+$this->assertFalse($selectionList->isEmpty());44-$this->assertFalse($selectionList2->isEmpty());45 }85 }468647-public function testIsStaleChunkSelected(): void87+public function testMarkStaleAndIsStaleChunkSelected(): void48 {88 {49-$selectionList = new SelectionList([], [], true);89+$selectionList = new SelectionList();50-$this->assertTrue($selectionList->isStaleChunkSelected());90+91+$this->assertFalse($selectionList->isStaleChunkSelected());92+93+$selectionList->markStale();519452-$selectionList2 = new SelectionList([], [], false);95+$this->assertTrue($selectionList->isStaleChunkSelected());53-$this->assertFalse($selectionList2->isStaleChunkSelected());54 }96 }559756public function testIsFullChunk(): void98public function testIsFullChunk(): void57 {99 {58-$chunkSize = 200;100+$selectionList = new SelectionList();59-101+$selectionList->setChunkSize(self::CHUNK_SIZE);60-$fullList = range(1, $chunkSize);102+61-$selectionList = new SelectionList(103+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, self::CHUNK_SIZE));62- priorityEntityIdList: $fullList,104+63- normalEntityIdList: [],105+$this->assertTrue($selectionList->isFullChunk());64- chunkSize: $chunkSize,106+ }65- );107+108+public function testIsNotFullChunkWhenBelowChunkSize(): void109+ {110+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);111+112+$selectionList = new SelectionList();113+$selectionList->setChunkSize(self::CHUNK_SIZE);114+115+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range(1, $halfRange));116+117+$this->assertFalse($selectionList->isFullChunk());118+ }119+120+public function testIsFullChunkAcrossMultiplePriorityGroups(): void121+ {122+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);123+124+$selectionList = new SelectionList();125+$selectionList->setChunkSize(self::CHUNK_SIZE);126+127+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, $halfRange));128+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range($halfRange + 1, self::CHUNK_SIZE));129+130+$this->assertTrue($selectionList->isFullChunk());131+ }132+133+public function testSetChunkSizeAffectsIsFullChunk(): void134+ {135+$selectionList = new SelectionList();136+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [1, 2, 3]);137+138+$selectionList->setChunkSize(3);66$this->assertTrue($selectionList->isFullChunk());139$this->assertTrue($selectionList->isFullChunk());6714068-$halfRange = (int) ceil($chunkSize / 2);141+$selectionList->setChunkSize(50);69-142+$this->assertFalse($selectionList->isFullChunk());70-$notFullList = range(1, $halfRange);71-$selectionList2 = new SelectionList(72- priorityEntityIdList: $notFullList,73- normalEntityIdList: [],74- chunkSize: $chunkSize,75- );76-$this->assertFalse($selectionList2->isFullChunk());77-78-$priorityRange = range(1, $halfRange);79-$normalRange = range($halfRange + 1, $chunkSize);80-81-$selectionList3 = new SelectionList(82- priorityEntityIdList: $priorityRange,83- normalEntityIdList: $normalRange,84- chunkSize: $chunkSize,85- );86-$this->assertTrue($selectionList3->isFullChunk());87 }143 }88}144}</selection>” selected. Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
You said
I’m on page “<tabTitle>JY-20808 low priority indexing queue by Vasil-Jimi</tabTitle>” with “<selection>@@ -5,84 +5,140 @@5namespace Tests\Unit\Component\ES\Processor\DTOs;5namespace Tests\Unit\Component\ES\Processor\DTOs;667use Jiminny\Component\ES\Processor\DTOs\SelectionList;7use Jiminny\Component\ES\Processor\DTOs\SelectionList;8+use Jiminny\Component\ES\QueuePriorityEnum;8use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\Attributes\CoversClass;9use PHPUnit\Framework\TestCase;10use PHPUnit\Framework\TestCase;101111#[CoversClass(SelectionList::class)]12#[CoversClass(SelectionList::class)]12final class SelectionListTest extends TestCase13final class SelectionListTest extends TestCase13{14{14-public function testGetEntityIds(): void15+private const int CHUNK_SIZE = 100;16+17+public function testAddEntityListAndGetAllIds(): void18+ {19+$selectionList = new SelectionList();20+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);21+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);22+23+$this->assertSame([1, 2, 3, 4], $selectionList->getAllIds());24+ }25+26+public function testAddEntityListMergesMultipleCallsForSamePriority(): void27+ {28+$selectionList = new SelectionList();29+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);30+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [3, 4]);31+32+$this->assertSame([1, 2, 3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));33+ }34+35+public function testGetEntityListItemsByGroup(): void36+ {37+$selectionList = new SelectionList();38+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1, 2]);39+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [3, 4]);40+$selectionList->addEntityList(QueuePriorityEnum::LOW, [5, 6]);41+42+$this->assertSame([1, 2], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));43+$this->assertSame([3, 4], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::NORMAL));44+$this->assertSame([5, 6], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::LOW));45+ }46+47+public function testGetEntityListItemsByGroupReturnsEmptyArrayForUnknownGroup(): void15 {48 {16-$priority = ['p1', 'p2'];49+$selectionList = new SelectionList();17-$normal = ['n1', 'n2'];50+18-$selectionList = new SelectionList($priority, $normal);51+$this->assertSame([], $selectionList->getEntityListItemsByGroup(QueuePriorityEnum::HIGH));52+ }195320-$this->assertSame($priority, $selectionList->getHighPriorityEntityIds());54+public function testGetEntityListGroups(): void21-$this->assertSame($normal, $selectionList->getNormalPriorityEntityId());55+ {56+$selectionList = new SelectionList();57+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);58+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2]);225923-$this->assertSame(['p1', 'p2', 'n1', 'n2'], $selectionList->getAllIds());60+$this->assertSame(['high', 'normal'], $selectionList->getEntityListGroups());24 }61 }256226public function testCountReturnsTotalCount(): void63public function testCountReturnsTotalCount(): void27 {64 {28-$selectionList = new SelectionList(['p1'], ['n1', 'n2']);65+$selectionList = new SelectionList();66+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);67+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [2, 3]);68+29$this->assertSame(3, $selectionList->count());69$this->assertSame(3, $selectionList->count());30 }70 }317132-public function testIsEmptyReturnsTrueWhenBothListsAreEmpty(): void72+public function testIsEmptyReturnsTrueWhenNoEntitiesAdded(): void33 {73 {34-$selectionList = new SelectionList([], []);74+$selectionList = new SelectionList();75+35$this->assertTrue($selectionList->isEmpty());76$this->assertTrue($selectionList->isEmpty());36 }77 }377838-public function testIsEmptyReturnsFalseWhenAnyListIsNotEmpty(): void79+public function testIsEmptyReturnsFalseWhenEntitiesAdded(): void39 {80 {40-$selectionList = new SelectionList(['p1'], []);81+$selectionList = new SelectionList();41-$this->assertFalse($selectionList->isEmpty());82+$selectionList->addEntityList(QueuePriorityEnum::HIGH, [1]);428343-$selectionList2 = new SelectionList([], ['n1']);84+$this->assertFalse($selectionList->isEmpty());44-$this->assertFalse($selectionList2->isEmpty());45 }85 }468647-public function testIsStaleChunkSelected(): void87+public function testMarkStaleAndIsStaleChunkSelected(): void48 {88 {49-$selectionList = new SelectionList([], [], true);89+$selectionList = new SelectionList();50-$this->assertTrue($selectionList->isStaleChunkSelected());90+91+$this->assertFalse($selectionList->isStaleChunkSelected());92+93+$selectionList->markStale();519452-$selectionList2 = new SelectionList([], [], false);95+$this->assertTrue($selectionList->isStaleChunkSelected());53-$this->assertFalse($selectionList2->isStaleChunkSelected());54 }96 }559756public function testIsFullChunk(): void98public function testIsFullChunk(): void57 {99 {58-$chunkSize = 200;100+$selectionList = new SelectionList();59-101+$selectionList->setChunkSize(self::CHUNK_SIZE);60-$fullList = range(1, $chunkSize);102+61-$selectionList = new SelectionList(103+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, self::CHUNK_SIZE));62- priorityEntityIdList: $fullList,104+63- normalEntityIdList: [],105+$this->assertTrue($selectionList->isFullChunk());64- chunkSize: $chunkSize,106+ }65- );107+108+public function testIsNotFullChunkWhenBelowChunkSize(): void109+ {110+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);111+112+$selectionList = new SelectionList();113+$selectionList->setChunkSize(self::CHUNK_SIZE);114+115+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range(1, $halfRange));116+117+$this->assertFalse($selectionList->isFullChunk());118+ }119+120+public function testIsFullChunkAcrossMultiplePriorityGroups(): void121+ {122+$halfRange = (int) ceil(self::CHUNK_SIZE / 2);123+124+$selectionList = new SelectionList();125+$selectionList->setChunkSize(self::CHUNK_SIZE);126+127+$selectionList->addEntityList(QueuePriorityEnum::HIGH, range(1, $halfRange));128+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, range($halfRange + 1, self::CHUNK_SIZE));129+130+$this->assertTrue($selectionList->isFullChunk());131+ }132+133+public function testSetChunkSizeAffectsIsFullChunk(): void134+ {135+$selectionList = new SelectionList();136+$selectionList->addEntityList(QueuePriorityEnum::NORMAL, [1, 2, 3]);137+138+$selectionList->setChunkSize(3);66$this->assertTrue($selectionList->isFullChunk());139$this->assertTrue($selectionList->isFullChunk());6714068-$halfRange = (int) ceil($chunkSize / 2);141+$selectionList->setChunkSize(50);69-142+$this->assertFalse($selectionList->isFullChunk());70-$notFullList = range(1, $halfRange);71-$selectionList2 = new SelectionList(72- priorityEntityIdList: $notFullList,73- normalEntityIdList: [],74- chunkSize: $chunkSize,75- );76-$this->assertFalse($selectionList2->isFullChunk());77-78-$priorityRange = range(1, $halfRange);79-$normalRange = range($halfRange + 1, $chunkSize);80-81-$selectionList3 = new SelectionList(82- priorityEntityIdList: $priorityRange,83- normalEntityIdList: $normalRange,84- chunkSize: $chunkSize,85- );86-$this->assertTrue($selectionList3->isFullChunk());87 }143 }88}144}</selection>” selected.
Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
Expand
Listen
Gemini said
Gemini said
Here is a precise and concise summary of the code selection, which refactors
SelectionListTest
to accommodate a shift from fixed high/normal priority properties to a more dynamic, enum-driven priority group architecture.
Refactored Priority Handling
Refactored Priority Handling
Dynamic Priority Groups:
Replaces fixed high/normal array parameters with the
QueuePriorityEnum
(
HIGH
,
NORMAL
,
LOW
), allowing entity lists to be managed dynamically by group types.
Merged Registrations:
Validates that multiple
addEntityList()
calls targeting the same priority level properly merge their data instead of overwriting it.
Group Querying:
Tests new methods
getEntityListItemsByGroup()
and
getEntityListGroups()
to retrieve group-specific IDs or an array of active group names (e.g.,
['high', 'normal']
).
Updated State and Count Validations
Updated State and Count Validations
Instantiation:
Tests now instantiate
SelectionList
without initial constructor arguments, using setters and action methods instead.
Empty States:
Updates assertions (
isEmpty()
) to verify state based on whether any entities have been dynamically added, rather than checking pre-defined constructor arrays.
Stale Flags:
Changes from passing a
stale
boolean via the constructor to explicitly triggering a mutable
$selectionList->markStale()
method.
Chunk Capacity Behavior
Chunk Capacity Behavior
Flexible Chunk Sizes:
Introduces a
setChunkSize()
method, replacing the constructor-configured chunk boundary.
Capacity Assertions:
Retains and updates the logic for
isFullChunk()
, ensuring it accurately evaluates capacity when elements span multiple dynamic priority groups or when the chunk limit is altered mid-lifecycle.
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
You said Are there any issues or potential bolltenecks
You said
Are there any issues or potential bolltenecks
Listen
Show thinking
Gemini said
Gemini said
Based on the test diff you provided, we can infer a few potential issues and architectural bottlenecks in the underlying
SelectionList
implementation. While the tests are well-structured, the behavior they describe hints at a few areas you might want to keep an eye on, particularly for an Elasticsearch (ES) processor queue.
Here are the potential issues and bottlenecks to consider:
1. Array Merging Overhead (Performance)
1. Array Merging Overhead (Performance)
The
testAddEntityListMergesMultipleCallsForSamePriority
test shows that calling
addEntityList()
multiple times for the same priority group merges the arrays.
The Bottleneck:
If the underlying implementation uses
array_merge()
inside a loop to append these IDs, it creates a new array in memory every time it's called. If you are processing thousands of entities and appending them in small batches, this can lead to memory bloat and CPU degradation due to constant array reallocation.
Mitigation:
Ensure the implementation handles batch appends efficiently (e.g., using the spread operator
[...$existing, ...$new]
or appending via references) or groups the arrays before doing a single merge.
2. Lack of Uniqueness/Deduplication (Logic)
2. Lack of Uniqueness/Deduplication (Logic)
None of the tests explicitly verify what happens if duplicate IDs are added. For example, if you call
addEntityList(QueuePriorityEnum::HIGH, [1, 2])
and then
addEntityList(QueuePriorityEnum::HIGH, [2, 3])
.
The Issue:
Does the getter return
[1, 2, 2, 3]
or
[1, 2, 3]
? If duplicates are not filtered out, your ES processor might end up indexing the same entity multiple times in a single chunk, which wastes I/O and processor cycles.
Mitigation:
The underlying class should ideally enforce uniqueness (e.g., using
array_unique()
before resolving the chunk, or using ID keys as associative array keys to naturally prevent duplicates).
3. Memory Consumption with Large Chunks (Scalability)
3. Memory Consumption with Large Chunks (Scalability)
The tests introduce
setChunkSize()
. If this chunk size is configured to be very large (e.g., tens of thousands of IDs to bulk-send to Elasticsearch).
The Bottleneck:
Holding massive arrays of IDs in memory across different priority groups can quickly hit PHP memory limits, especially if this
SelectionList
object is kept alive during a long-running daemon or cron job.
Mitigation:
Monitor memory usage if
CHUNK_SIZE
grows, and ensure the
SelectionList
object is properly destroyed (or garbage collected) after the chunk is processed to free up memory.
4. Type Juggling
4. Type Juggling
In the original code, string IDs were used (
['p1', 'p2']
). In the refactored tests, integer IDs are used (
[1, 2]
).
The Issue:
If your system handles UUIDs (strings) vs. relational DB auto-increments (integers), make sure the underlying implementation uses strict comparisons (
===
) where necessary, especially if deduplicating arrays, to prevent PHP from juggling
'1'
and
1
improperly.
Would you like me to draft a quick test case to check how the class currently handles duplicate IDs, or is deduplication intentionally handled further down your pipeline?
Good response
Bad response
Share & export
Copy
Show more options
Copy prompt
Edit
You said what about non test files
You said
what about non test files
Show thinking
Gemini said
Gemini said
Since we don't have the actual
SelectionList.php
file,
we have to reverse-engineer its implementation based on the test behaviors.
Moving from a constructor-based,
fixed-property model to a dynamic,
mutable model introduces several potential pitfalls in the implementation file itself.
Here are the specific bottlenecks and issues to watch out for in the
SelectionList.php
source code:
1. The count() Method Implementation (Performance Trap)
1. The
count()
Method Implementation (Performance Trap)
The Risk:
If
count()
is implemented by calling
$this->getAllIds()
and counting the resulting merged array (e.
g.,
return count($this->getAllIds());
),
it will be highly inefficient.
It would allocate a brand-new array in memory just to determine its length.
The Fix:
count()
should mathematically sum the sizes of the underlying group arrays without merging them:
return array_sum(array_map('count', $this->entityLists));
2. Loss of Immutability (State Management)
2. Loss of Immutability (State Management)
The Risk:
The original code passed all data via the constructor,
implying the DTO (Data Transfer Object) was immutable once created.
The refactored version introduces mutability (
addEntityList()
,
setChunkSize()
,
markStale()
).
DTOs are typically passed between architectural layers; if one layer modifies the chunk size or adds entities while another is processing it,
it can cause unpredictable side effects or race conditions in your queue processor.
The Fix:
Ensure that this object is not shared across asynchronous contexts,
or consider returning cloned instances when mutating state if strict immutability is required.
3. Enum Array Key Violations (PHP Quirks)
3. Enum Array Key Violations (PHP Quirks)
The Risk:
The tests show
QueuePriorityEnum::HIGH
being passed to methods.
If the underlying code attempts to use this Enum directly as an array key (e.
g.,
$this->lists[$priority] = $ids;
),
PHP will throw an error because objects (including Enums) cannot be used as standard array keys.
The Fix:
The implementation must map the Enum to its underlying scalar value (e.
g.,
$this->lists[$priority->value]
) or use PHP's
WeakMap
or
SplObjectStorage
,
though standard arrays using
$priority->value
are much faster for simple lists.
4. Uninitialized State Defaults (Logic Errors)
4. Uninitialized State Defaults (Logic Errors)
The Risk:
The tests instantiate
new SelectionList()
without arguments,
then later call
$selectionList->setChunkSize(100)...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082/changes
|
55649
|
|
55758
|
FirefoxFileEditViewHistoryBookmarksProfilesToolsWi FirefoxFileEditViewHistoryBookmarksProfilesToolsWindowHelp> 0.Daily - Platform • now100% L2Tue 19 May 9:48:23meet.google.com/agt-teir-cwt?authuser=lukas.kovalik%40jiminny.com+19Nikolay Yankov (Presenting)(7 Service-Attention() Scheml© Cursor -X * Caude|https://fiminny.atlassian.net/jra/software/c/projects/JJY/boards/37?selectedissue=JY-2091923 SSHDatadog ©e CauờiPlatform Team %.Q Search boards00PJr-ToL1 38 JT205LATUS Group - Exec Summary InaccuracyNotify the user if awotat0omddeleted but is used in AJ..AJREPORTSBacklogД Jy-200763 ..=©~ Koy detailsDescriptionEdit descriptionSteps to reproduceldeleted but is used in AJnewertAJREPORTSBacklogД JY-206152.5 •000 = 1Norelnetud outcomieNOO LUXTExpected outcomeAdd textUpgrade BE Ibraries - MayMAINTENANCSSubtasksAdd subtaskE JY-19958•**=©Linked work itemshelinkeewetsuggestionsAUTO-DETECTED AGTIVITY TYPBacklogA JY-204702 *++=~ ActivityAotcontnSuggest a reply.-Can I get more info..?Status update.Tue 19 May 9:48CostiC X aco.de XAlowor X3 Pioeline:© JminnL tasкnsignts & Coachn.© DevL Al Bookmarxs@ 1.In Dev vI Improve Bug~ DetailsAssignee3 Steliyan GeorgievAssign to meReporterf2 Stellyan GeorgievDevelopmentQ Open with VS Code1 branch4 commits1 pull request4 buildsthoueMEROEDReleases7 See all deploymentsLabeltNoneSub-ProductAdd optionsGroup: QueriesAJ Panorama for CallScoring in ODAUTOMATED AL SCORINGDeployedP J-20301 05[HubSpot) Optimise CRMrematching on deietenuospos eccounsrcomnactsPCATFORM STARILITYDeployedXE-JY-2072515 1=(Deadline 25 May) Migratdepricated Gemini 3.1 FleSetup test coverage for©J-1051 /0 " •**= 1Upgrade Python andNikolay YankovNikolay Ivanov2 othersSteliyan Georgiev9:48 AM | Daily - PlatformLukas Kovalik3:03Lộ3...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082#pullrequestrevie github.com/jiminny/app/pull/12082#pullrequestreview-4316426636...
|
55758
|
|
55759
|
ActivityMoreSlackcalVIewJiminny...# platform-ticke ActivityMoreSlackcalVIewJiminny...# platform-tickets# product_launchesac random# releases# sofia-office# support# thank-yous# the_people_of jimi...•? Direct messagesG. Vasil Vasilev. Galya Dimitrova ER. Aneliya Angelova. Nikolay Yankov. Stefka Stoyanova8. Stoyan TomovCa Todor Stamatov "9Mario Georgiey.. Nikolay Ivanov2o James GrahamRo Stoyan TanevR. Steliyan Georgiev =f Petko Kashinski. Lukas Kovalik y...i AppsG Jira Cloud8 ToastmistonWindowhelpQ Describe what you are looking forC. Vasil Vasilev• Messagest Add canvasUr FilesMoreполучавам Yesterday • икации отtoast (editedLukas Kovalik 10.39 AMVasil Vasilev 10:40 AMScreenshot 2026-05-18 at 10.40.05.png ~••• Xed (e.g., inside isEmpty() , isStaleChunkSelected().ap and array_sum every single time creates$totalCount = 0; property. Increment it insidet += count (SentityIds); . Then your count()->totalCount;.l are doing sequential round-trips to Redis for eachне знам лали го ползваш, но е многополеzен тwwnYasi Vasiley 9.18 AMЛобго утро. Лукашкогато имаш лнес възможностмоля те погледни тоя ПР.https://github.com/iminny/app/pull/1z0ozMessage Vasil Vasilev1...) (Network trip)AL.. Network trio)aintains existing behavior, sequential Redis calls block-comes an issue, consider migrating these extractiong Pipeline/Transactions to fetch multiple queues in a+ Aa €thods like-Update across the entire codebase have been updatedto pass tne new cnum instead or a boorean, or should we add a temporary fallback to support bothtypes during the transition?• Enter a prompt for GeminiProvYour Jiminny chats aren't used to improve our models. Gemini is Al. It can make mistakes, so double check it. Your privacy & GeminiSummarize page. 40 1 Daily - Platform - now БF 100% C/ & • Tue 19 May 9:48:238? OpenJY-20808 low priority indexing queue #12082Vasil-Jiminny wants to merge 36 commits into master from JY-20808-low-priority-indexing-queue~ Quality Gate passedIssuesv O New issues@ 0 Accepted issuesMeasuresv O Security Hotspots100.0% Coverage on New Codev 0.0% Duplication on New CodeSee analysis details on SonarQube CloudLakvlak aooroved these chanaes nowThis branch has not been deployedNo denlovmentsChanges approved1 aporovina review bv reviewers with write access.V1 aporoval>8 2 pending reviews >O All checks have passed1 skinned 12 successful checksA This branch is out-of-date with the base branchMerge the latest changes from master into this branch. This merge commit will beassociated with LakvLakEnable auto-mergeYou can also merge this with the command line. View command line instructions.Add a commentWritePreviewAdd your comment here…..View reviewed changesUpdate branchStill in oroaress? Convert to drafi...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082#pullrequestrevie github.com/jiminny/app/pull/12082#pullrequestreview-4316426636...
|
55759
|
|
55637
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
github.com/jiminny/app/pull/12082
github.com/jiminny/app/pull/12082
Close tab
New Tab
Customize sidebar
Open Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
Waiting for github.com…...
|
Firefox
|
Work — Mozilla Firefox
|
github.com/jiminny/app/pull/12082
|
55637
|
|
55638
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Open Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
Transferring data from github.com…...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082
|
55638
|
|
55639
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20725 add HS rate limit handling on activities rematching by LakyLak · Pull Request #12066 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app
Close tab
New Tab
Customize sidebar
Open Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
Skip to content
Skip to content
Open menu
Homepage (g then d)
jiminny
jiminny
app
app
Search or jump to…
Type
/
to search
Chat with Copilot
Open Copilot…
Create new...
All issues(g then i)
All pull requests
All repositories
You have unread notifications(g then n)
Open user navigation menu
Repository navigation
Repository navigation
Code
Code
Pull requests (31)
Pull requests
(
31
)
Agents
Agents
Actions
Actions
Wiki...
|
Firefox
|
JY-20808 low priority indexing queue by Vasil-Jimi JY-20808 low priority indexing queue by Vasil-Jiminny · Pull Request #12082 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12082
|
55639
|
|
41284
|
FirefoxcalVIewmistonbookmarksProtllesToolsWindowcl FirefoxcalVIewmistonbookmarksProtllesToolsWindowclnuo.com/minnyapp/oull/140/cmele# Support Daily - in 1h 43 m100% L2. Thu 14 May 13:18:01Platform Sprint 4 Q2 - Platform TeW Usage I Windsurf[SRD-6848] Sidekick SMS issue(JY-20891] Sidekick SMS issueJY-20891 add support for secon47 ISRD-68491 Recorded call does8 Jiminn8 Jiminny≤7 ISRD-68531 Moyso - Potential de[JY-20903] Recorded call does n@JY-20903 Update activity stageConfiaure SSH accass to multinle* Useful commands - Engineering -8 Dey Tools - Elastic9 liminnvCloudWatch | eu-west-1C CloudWatch I eu-west-1ACioudWatch ton-woct.≤7 (uy-209041 Fix UodateActivitvElagithub.com/fiminny/app/pull/12L New TalTransferring data from aithub.com....
|
Firefox
|
JY-20904 Fix UpdateActivityElasticSearchDocumentCo JY-20904 Fix UpdateActivityElasticSearchDocumentCommand find activity… by LakyLak · Pull Request #12078 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12078
|
41284
|
|
41285
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Usage | Windsurf
Usage | Windsurf
Unnamed Group
[SRD-6848] Sidekick SMS issue - Jira
[SRD-6848] Sidekick SMS issue - Jira
[JY-20891] Sidekick SMS issue - Jira
[JY-20891] Sidekick SMS issue - Jira
JY-20891 add support for secondary email by LakyLak · Pull Request #12073 · jiminny/app
JY-20891 add support for secondary email by LakyLak · Pull Request #12073 · jiminny/app
Unnamed Group
[SRD-6849] Recorded call does not appear on the dashboard - Jira
[SRD-6849] Recorded call does not appear on the dashboard - Jira
Jiminny
Jiminny
Jiminny
Jiminny
Jiminny
Jiminny
Unnamed Group
[SRD-6853] Moxso - Potential deal stages bug - Jira
[SRD-6853] Moxso - Potential deal stages bug - Jira
[JY-20903] Recorded call does not appear on the dashboard - Jira
[JY-20903] Recorded call does not appear on the dashboard - Jira
JY-20903 Update activity stage when opportunity is changed by LakyLak · Pull Request #12077 · jiminny/app
JY-20903 Update activity stage when opportunity is changed by LakyLak · Pull Request #12077 · jiminny/app
Configure SSH access to multiple environment - Engineering - Confluence
Configure SSH access to multiple environment - Engineering - Confluence
Useful commands - Engineering - Confluence
Useful commands - Engineering - Confluence
Dev Tools - Elastic
Dev Tools - Elastic
Jiminny
Jiminny
CloudWatch | eu-west-1
CloudWatch | eu-west-1
CloudWatch | eu-west-1
CloudWatch | eu-west-1
CloudWatch | eu-west-1
CloudWatch | eu-west-1
Unnamed Group
[JY-20904] Fix UpdateActivityElasticSearchDocumentCommand - Jira
[JY-20904] Fix UpdateActivityElasticSearchDocumentCommand - Jira
JY-20904 Fix UpdateActivityElasticSearchDocumentCommand find activity… by LakyLak · Pull Request #12078 · jiminny/app
JY-20904 Fix UpdateActivityElasticSearchDocumentCommand find activity… by LakyLak · Pull Request #12078 · jiminny/app
Close tab
New Tab
Customize sidebar
Open Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
Skip to content
Skip to content
Open menu
Homepage (g then d)
jiminny
jiminny
app
app
Search or jump to…
Type
/
to search
Chat with Copilot
Open Copilot…
Create new...
All issues(g then i)
All pull requests
All repositories
You have no unread notifications(g then n)
Open user navigation menu
Repository navigation
Repository navigation
Code
Code
Pull requests (29)
Pull requests
(
29
)
Agents
Agents
Actions
Actions
Wiki
Wiki
Security and quality
Security and quality
Insights
Insights
Settings
Settings
Important update
Important update
On April 24 we'll start using GitHub Copilot interaction data for AI model training unless you opt out.
Review this update
Review this update
and manage your preferences in your
GitHub account settings
GitHub account settings
.
Dismiss banner
JY-20904 Fix UpdateActivityElasticSearchDocumentCommand find activity… #12078 Edit title
JY-20904 Fix UpdateActivityElasticSearchDocumentCommand find activity…
#
12078
Edit title
View statusView status
Loading
Loading merge status
Code
Code
Open
LakyLak
LakyLak
wants to merge 1 commit into
master
master
from
JY-20904-fix-update-es-on-activity-command
JY-20904-fix-update-es-on-activity-command
Copy head branch name to clipboard
Conversation (0)
Conversation
(
0
)
Commits (1)
Commits
(
1
)
Checks (2)
Checks
(
2
)
Files changed (1)
Files changed
(
1
)
Conversation
Conversation
@LakyLak
Show options
LakyLak commented now
LakyLak
LakyLak
commented
now
now
… query
JIRA: JY-20904
JIRA:
JY-20904
JY-20904
Changes:
Changes:
Fix command activity search
Add or remove reactions
@LakyLak
JY-20904
JY-20904
Fix UpdateActivityElasticSearchDocumentCommand find activity…
Fix UpdateActivityElasticSearchDocumentCommand find activity…
…
28f87e1
28f87e1
This branch has not been deployed
This branch has not been deployed
No deployments
Loading
@LakyLak
Add a comment
Add a comment
Comment
Write
Write
Preview
Preview
Comment
Markdown is supported
Markdown
is supported
Paste, drop, or click to add files
Paste, drop, or click to add files
Close pull request
Close pull request
Comment
Remember, contributions to this repository should follow our
GitHub Community Guidelines
GitHub Community Guidelines
.
ProTip!
Add comments to specific lines under
Files changed
Files changed...
|
Firefox
|
JY-20904 Fix UpdateActivityElasticSearchDocumentCo JY-20904 Fix UpdateActivityElasticSearchDocumentCommand find activity… by LakyLak · Pull Request #12078 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12078
|
41285
|
|
41286
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Usage | Windsurf
Usage | Windsurf
Unnamed Group
[SRD-6848] Sidekick SMS issue - Jira
[SRD-6848] Sidekick SMS issue - Jira
[JY-20891] Sidekick SMS issue - Jira
[JY-20891] Sidekick SMS issue - Jira
JY-20891 add support for secondary email by LakyLak · Pull Request #12073 · jiminny/app
JY-20891 add support for secondary email by LakyLak · Pull Request #12073 · jiminny/app
Unnamed Group
[SRD-6849] Recorded call does not appear on the dashboard - Jira
[SRD-6849] Recorded call does not appear on the dashboard - Jira
Jiminny
Jiminny
Jiminny
Jiminny
Jiminny
Jiminny
Unnamed Group
[SRD-6853] Moxso - Potential deal stages bug - Jira
[SRD-6853] Moxso - Potential deal stages bug - Jira
[JY-20903] Recorded call does not appear on the dashboard - Jira
[JY-20903] Recorded call does not appear on the dashboard - Jira
JY-20903 Update activity stage when opportunity is changed by LakyLak · Pull Request #12077 · jiminny/app
JY-20903 Update activity stage when opportunity is changed by LakyLak · Pull Request #12077 · jiminny/app
Configure SSH access to multiple environment - Engineering - Confluence
Configure SSH access to multiple environment - Engineering - Confluence
Useful commands - Engineering - Confluence
Useful commands - Engineering - Confluence
Dev Tools - Elastic
Dev Tools - Elastic
Jiminny
Jiminny
CloudWatch | eu-west-1
CloudWatch | eu-west-1
CloudWatch | eu-west-1
CloudWatch | eu-west-1
CloudWatch | eu-west-1
CloudWatch | eu-west-1
Unnamed Group
[JY-20904] Fix UpdateActivityElasticSearchDocumentCommand - Jira
[JY-20904] Fix UpdateActivityElasticSearchDocumentCommand - Jira
JY-20904 Fix UpdateActivityElasticSearchDocumentCommand find activity… by LakyLak · Pull Request #12078 · jiminny/app
JY-20904 Fix UpdateActivityElasticSearchDocumentCommand find activity… by LakyLak · Pull Request #12078 · jiminny/app
Close tab
New Tab
Customize sidebar
Open Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
Skip to content
Skip to content
Open menu
Homepage (g then d)
jiminny
jiminny
app
app
Search or jump to…
Type
/
to search
Chat with Copilot
Open Copilot…
Create new...
All issues(g then i)
All pull requests
All repositories
You have unread notifications(g then n)
Open user navigation menu
Repository navigation
Repository navigation
Code
Code
Pull requests (29)
Pull requests
(
29
)
Agents
Agents
Actions
Actions
Wiki
Wiki
Security and quality
Security and quality
Insights
Insights
Settings
Settings
Important update
Important update
On April 24 we'll start using GitHub Copilot interaction data for AI model training unless you opt out.
Review this update
Review this update
and manage your preferences in your
GitHub account settings
GitHub account settings
.
Dismiss banner
JY-20904 Fix UpdateActivityElasticSearchDocumentCommand find activity… #12078 Edit title
JY-20904 Fix UpdateActivityElasticSearchDocumentCommand find activity…
#
12078
Edit title
Checks pending
Checks pending
Code
Code
Open
LakyLak
LakyLak
wants to merge 1 commit into
master
master
from
JY-20904-fix-update-es-on-activity-command
JY-20904-fix-update-es-on-activity-command
Copy head branch name to clipboard
Lines changed: 9 additions & 1 deletion
Conversation (0)
Conversation
(
0
)
Commits (1)
Commits
(
1
)
Checks (2)
Checks
(
2
)
Files changed (1)
Files changed
(
1
)
Conversation
Conversation
@LakyLak...
|
Firefox
|
JY-20904 Fix UpdateActivityElasticSearchDocumentCo JY-20904 Fix UpdateActivityElasticSearchDocumentCommand find activity… by LakyLak · Pull Request #12078 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12078
|
41286
|
|
41287
|
Platform Sprint 4 Q2 - Platform Team - Scrum Board Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 4 Q2 - Platform Team - Scrum Board - Jira
Usage | Windsurf
Usage | Windsurf
Unnamed Group
[SRD-6848] Sidekick SMS issue - Jira
[SRD-6848] Sidekick SMS issue - Jira
[JY-20891] Sidekick SMS issue - Jira
[JY-20891] Sidekick SMS issue - Jira
JY-20891 add support for secondary email by LakyLak · Pull Request #12073 · jiminny/app
JY-20891 add support for secondary email by LakyLak · Pull Request #12073 · jiminny/app
Unnamed Group
[SRD-6849] Recorded call does not appear on the dashboard - Jira
[SRD-6849] Recorded call does not appear on the dashboard - Jira
Jiminny
Jiminny
Jiminny
Jiminny
Jiminny
Jiminny
Unnamed Group
[SRD-6853] Moxso - Potential deal stages bug - Jira
[SRD-6853] Moxso - Potential deal stages bug - Jira
[JY-20903] Recorded call does not appear on the dashboard - Jira
[JY-20903] Recorded call does not appear on the dashboard - Jira
JY-20903 Update activity stage when opportunity is changed by LakyLak · Pull Request #12077 · jiminny/app
JY-20903 Update activity stage when opportunity is changed by LakyLak · Pull Request #12077 · jiminny/app
Configure SSH access to multiple environment - Engineering - Confluence
Configure SSH access to multiple environment - Engineering - Confluence
Useful commands - Engineering - Confluence
Useful commands - Engineering - Confluence
Dev Tools - Elastic
Dev Tools - Elastic
Jiminny
Jiminny
CloudWatch | eu-west-1
CloudWatch | eu-west-1
CloudWatch | eu-west-1
CloudWatch | eu-west-1
CloudWatch | eu-west-1
CloudWatch | eu-west-1
Unnamed Group
[JY-20904] Fix UpdateActivityElasticSearchDocumentCommand - Jira
[JY-20904] Fix UpdateActivityElasticSearchDocumentCommand - Jira
JY-20904 Fix UpdateActivityElasticSearchDocumentCommand find activity… by LakyLak · Pull Request #12078 · jiminny/app
JY-20904 Fix UpdateActivityElasticSearchDocumentCommand find activity… by LakyLak · Pull Request #12078 · jiminny/app
Close tab
New Tab
Customize sidebar
Open Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
Skip to content
Skip to content
Open menu
Homepage (g then d)
jiminny
jiminny
app
app
Search or jump to…
Type
/
to search
Chat with Copilot
Open Copilot…
Create new...
All issues(g then i)
All pull requests
All repositories
You have unread notifications(g then n)
Open user navigation menu
Repository navigation
Repository navigation
Code
Code
Pull requests (29)
Pull requests
(
29
)
Agents
Agents
Actions
Actions
Wiki
Wiki
Security and quality
Security and quality
Insights
Insights...
|
Firefox
|
JY-20904 Fix UpdateActivityElasticSearchDocumentCo JY-20904 Fix UpdateActivityElasticSearchDocumentCommand find activity… by LakyLak · Pull Request #12078 · jiminny/app — Work...
|
github.com/jiminny/app/pull/12078
|
41287
|