|
FirefoxFileEoitViewHistoryBookmarksProfilesToolsWi FirefoxFileEoitViewHistoryBookmarksProfilesToolsWindowHelpgithub.com/liminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab/dd0782d4a4d93075ca5ce05903d77e8Google GeminiJy 19798 evaluation for ai activity tvpes #468Platform Sprint 1 Q2 - Platform Tea( Jy 19798 evaluation for ai activ+ New lab8? OperAll commits-nikolaybiaivanov wants to merge 9 commits into master from JY-19798-evaluation-for-ai-activity-types Lsrc/eval/endpoints.pyE• Filter files...v = eval_ output/ai_activity_type/eval...v src/eval/eval-ai-activity-type.py L> • ground_truth..•aa-0.0 +1.330 ad>I inoutv F src/eval#endpoints.py+ eval-ai-activity-type.pyE .gitignoreDaily - Platform • now100%• Tue 14 Apr 9:50:33• 2 | 203 viewed|Awallind aporovali45 nannOI• Viewed+330 UUUUuVlewed+ #!/usr/bin/env python3+ 1mport asynclo+ import json+ import os+ trom pathlib import Path+ from typing import Any, Dict, List+ import aiohttp+ from src.component.call_processing.schemes.ai activity type_request importALACELVLEV voekequest11+ from src.component.call_processing.schemes.ai activity type_response importA1ACCIVICYIypekesponse+ trom src.eval.abstract_evaluator import Abstractevaluator+ from src.eval.collectors import collect test set as es collect test set+ trom src.eval.common_evaluator_runner import create_parser, run_evaluation+ from src.eval.endpoints import run endpoint as run endpoint http+ trom src,eval,models 1moort Evaluatzonkesult23Ask GeminiC A ESummarise page+ class AiActivityTypeEvaluator(AbstractEvaluator):ENDPOINT PATH = "call/ai-activity-type"REQUEST_MODEL = AiActivityTypeRequestRESrUNst MoDEL = ALACCIviLyIvpekesponseouTPUT_sUBFOLDER = "a1_activity_type"USES LLM EVALUATION = FalseREADS_FROM_DATASET_ FOLDER = TrueDATA SET: str = "initial"MINIMUM_REQUEST BODY_SIZE: int = 0SKLP_UNREPLAYABLE_ON_COLLECT: bool = TrueREPLAYABILITY CHECK TIMEOUT SECONDS: float = 20.0rurULAlL onuuNu Ikuin LlM. Dool = IalseGROUND TRUTH LIM ENDPOINT: str = "OPENAL:apt=5 2"LUMPAKE TU GKUUND IKuin: Str=esdeт 1nic selt, **kwaros. Any -> None:super().__init__(**kwargs)_repo_root = Path(_file_).parents (2]_datasets_root = _repo_root / "eval_output" / selt.OUTPUT_SUBFOLDER /"eval datasets"_dataset_dir = _datasets_root / selt.__class_.DATA_SElself._dataset_dir =_dataset_dirselt._inputs_air =_dataset_air / "Input"self. around truths dir= datacet dir "around truth'l...
|
NULL
|
NULL
|
NULL
|
8537
|
|
DiaFilelEditViewTabsBookmarksHistoryExtensionsWind DiaFilelEditViewTabsBookmarksHistoryExtensionsWindowHelp$0.| Daily - Platform • nowmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)Inbox (2) - niC 3y 19798 eval(7 Service-Deskjr Foet Ameson• Cost | Claude x© Actions secrC C Dependatinny/prophet/actions/runs/24195639151/ob/70624720594?pr=468B AWSE3 SSHEl Datados * Clude5 CircieclD Sentry8 TaskslActions secQ 8•Tue 14 Apr 9:50• Dependabot: xHa Dependatс *Insights & Coachin.• Dev• Ux0 Al BookmarksE:1Summary|Run Flake8 & MypyRun details© UsageWorkfiow fileRun Flake8 & Mypy•, Search logs IStefka StoyanovaHDSet up Python> O Install Poetry|> • Load cached Poetry dependencies |> © Install dependencies using Poetry|v @ Run Flake81 •Run poetry run flakeß, -count -select-E9,F63,F7,F82,N293,N291,W292 --show-source -statistics•/src/eval/eval-ai-activity-type.py:330:63: N292 no nevline at end of fileasyncio.runtrun_cvaluatzon(AlactavityTypetvaluator, args)Error: Process conpleted with exit code 1.• Run Mypy|• Post Load cached Poetry dependencies |• Post Set up Python|› © Post Checkout code> O Complete job• 2026 0i4u6, Inc.HortroPrivacy Security Status Community Docsvonest menesnDo not share my personal information)25s7san GeorgievKovalik9:50 AM | Daily - Platform5:118 • Tue 14 Apr 9:50:35Chat6Nikolay YankovNikolay Ivanov...
|
NULL
|
NULL
|
NULL
|
8538
|
|
firefoxFileEoitViewHistoryBookmarksProfilesToolsWi firefoxFileEoitViewHistoryBookmarksProfilesToolsWindow Helpgithub.com/liminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab/dd0782d4a4d93075ca5ce05903d77e8Google Gemini...Platform Sprint 1 Q2 - Platform Tea( Jy 19798 evaluation for ai activ+ New labim on page"<tab Itle>Jv19798evalarion tor allactivity types bynikol</tabTitle>"...Jy 19798 evaluation for ai activity tvpes #468i OperAll commits-nikolaybiaivanov wants to merge 9 commits into master from JY-19798-evaluation-for-ai-activity-types L• Filter files...src/eval/endpoints.pyEv = eval_ output/ai_activity_type/eval...v src/eval/eval-ai-activity-type.py L> • ground_truth..•aa-0.0 +1.330 ad>I inoutF src/eval= enapolnts.py+ eval-ai-activity-type.pyE .gitignoreDaily - Platform • now• 2 1 203 viewed|Awallind aporovali100%• Tue 14 Apr 9:50:3645 nannOI• Viewed+330 UUUUuVlewedInis Pytnon seript derines theAlActivltylypetvaluator class, aspecialized tool tor evaluating Al-arivenactivity type classification. It extends a baseevaluation namework to nanole datacollection, ground-truth comparison, andperformance scoringCore Functionality0 Enter a oromot for GeminiPro vYour Jiminny chats aren't used to improve our models. Gemini is Alane can make mistakes, Inciualne aoout peodleYour orivacy & GeminSummarise page+ #!/usr/bin/env python3+ 1mport asynclo+ import json+ import os+ trom pathlib import Path+ from typing import Any, Dict, List+ import aiohttp+ from src.component.call_processing.schemes.ai activity type_request importALACELVLEV voekequest11+ from src.component.call_processing.schemes.ai activity type_response importA1ACCIVICYIypekesponse+ trom src.eval.abstract_evaluator import Abstractevaluator+ from src.eval.collectors import collect test set as es collect test set+ trom src.eval.common_evaluator_runner import create_parser, run_evaluation+ from src.eval.endpoints import run endpoint as run endpoint http+ trom src,eval,models 1moort Evaluatzonkesult+ class AiActivityTypeEvaluator(AbstractEvaluator):ENDPOINT PATH = "call/ai-activity-type"REQUEST_MODEL = AiActivityTypeRequestRESrUNst MoDEL = ALACCIviLyIvpekesponse23ouTPUT_sUBFOLDER = "a1_activity_type"USES LLM EVALUATION = FalseREADS_FROM_DATASET_ FOLDER = TrueDATA SET: str = "initial"MINIMUM_REQUEST BODY_ SIZE: int = 0SKLP_UNREPLAYABLE_ON_COLLECT: bool = TrueREPLAYABILITY CHECK TIMEOUT SECONDS: float = 20.0rurULAlL onuuNu Ikuin LlM. Dool = IalseGROUND TRUTH LIM ENDPOINT: str = "OPENAL:apt=5 2"LUMPAKE TU GKUUND IKuin: Str=esdeт 1nic selt, **kwaros. Any -> None:super().__init__(**kwargs)_repo_root = Path(_file_).parents (2]_datasets_root = _repo_root / "eval_output" / selt.OUTPUT_SUBFOLDER /"eval datasets"_dataset_dir = _datasets_root / selt.__class__.DATA_SE!self._dataset_dir =_dataset_dirselt._inputs_air =_dataset_air / "Input"self. around truths dir= datacet dir "around truth'l...
|
NULL
|
NULL
|
NULL
|
8539
|
|
DiaFilelEditViewTabsBookmarksHistoryExtensionsWind DiaFilelEditViewTabsBookmarksHistoryExtensionsWindowHelp$0.| Daily - Platform • nowmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)Inbox (2) - niE:1Q 8•Tue 14 Apr 9:50C 3y 19798 eval(7 Service-Deskjr Foet Ameson• Cost | Claude x©Actions secrC C DependatActions sec• Dependabot: xHa Dependatinny/prophet/actions/runs/24195639151/ob/70624720594?pr=468B AWSE3 SSHс *El Datados * Clude5 CircieclD Sentry8 TaskslInsights & Coachin.• Dev• Ux.0 Al BookmarksRun Flake8 & Mypy•, Search logs ISummary|Run Flake8 & MypyRun details© UsageWorkfiow fileStefka StoyanovaHDSet up Python> O Install Poetry|> • Load cached Poetry dependencies |> © Install dependencies using Poetry|v @ Run Flake81 •Run poetry run flakeß, -count -select-E9,F63,F7,F82,N293,N291,W292 --show-source -statistics./src/eval/eval-al-activity-type.py:330:63: N292 no newline at end of fileasyncio.runtrun_cvaluatzon(AlactavityTypetvaluator, args)Error: Process conpleted with exit code 1.• Run Mypy|• Post Load cached Poetry dependencies |• Post Set up Python|› © Post Checkout code> O Complete job• 2026 0i4u6, Inc.HortroPrivacy Security Status Community Docsvonest menesn25san GeorgievKovalik9:50 AM | Daily - Platform5:148 • Tue 14 Apr 9:50:38Chat6Nikolay YankovNikolay Ivanov...
|
NULL
|
NULL
|
NULL
|
8540
|
|
DialFilelEditViewTabsBookmarksHistoryExtensionsWin DialFilelEditViewTabsBookmarksHistoryExtensionsWindowHelplall| Daily - Platform • nowmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)IEB)nbox (2) - nilC 3y 19798 eval(l Service-Desk xjr Foet Awesonx• Cost | Claude x@ Actions secrC C Dependathttps://github.comfjminny/prophet/actions/runs/24195639151/ob/70624720594?pr=468# AWS|E3 SSHEl Datados Clude3 CircieciD Sentry0 Tasksiminny / prophet&(> Code17 Pull requests• Actions85 Projectsa WildlSecurity and quality 20 z Insights ® Settings• Jy 19798 evaluation for ai activity types #1918|SummaryiA XOON|© Run Flake8 & MypyКи оcиilÔ Usage& Workflow fie> Annotations1 error and 1 warningRun Flake8 & Mypyfalled 5 days apo in 57s> O Setupjob> © Checkout code|> • Set up Python |> © Install Poetry|> • Load cached Poetry dependencies |> • Install dependencies using Poetry |Run Haked.1 •Run poetry run flakeß, -count -select-E9, F63,F7,F82,N293, N291,N292 -shov-source -statistics |/src/eval/eval-at-actavity-type.py:3301631 keyk no dorithe at cho of Taleasyncio.run(run_evaluation(AlActIvityTypeEvaluater, args))Error: Process conpleted with exit code 1.• Run Mypy |Post Load cached Poetry deoendençies.• Post Set up Python|• Actions sec:• Dependabot: xO Far Dependa:с:|Insights & Coachin.• DevDux.Q Type L] to search88Tue 14 Apr 9:500 Al BookmarksStefka StoyanovaRe-runjobs -Q Search logs|125Steliyan GeorgievKovalik9:50 AM | Daily - Platform5:238 • Tue 14 Apr 9:50:47Chat6Nikolay YankovNikolay Ivanov...
|
NULL
|
NULL
|
NULL
|
8541
|
|
JY-20543 add AJ reports User pilot tracking by Lak JY-20543 add AJ reports User pilot tracking by LakyLak · Pull Request #11932 · jiminny/app
Platform Sprint 1 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 1 Q2 - Platform Team - Scrum Board - Jira
Jy 19798 evaluation for ai activity types by nikolaybiaivanov · Pull Request #468 · jiminny/prophet
Jy 19798 evaluation for ai activity types by nikolaybiaivanov · Pull Request #468 · jiminny/prophet
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Copy prompt
You said I’m on page “<tabTitle>Jy 19798 evaluation for ai activity types by nikol</tabTitle>” with “<selection>@@ -0,0 +1,330 @@1+#!/usr/bin/env python32+import asyncio3+import json4+import os5+from pathlib import Path6+from typing import Any, Dict, List7+8+import aiohttp9+10+from src.component.call_processing.schemes.ai_activity_type_request import AiActivityTypeRequest11+from src.component.call_processing.schemes.ai_activity_type_response import AiActivityTypeResponse12+from src.eval.abstract_evaluator import AbstractEvaluator13+from src.eval.collectors import collect_test_set as es_collect_test_set14+from src.eval.common_evaluator_runner import create_parser, run_evaluation15+from src.eval.endpoints import run_endpoint as run_endpoint_http16+from src.eval.models import EvaluationResult17+18+19+class AiActivityTypeEvaluator(AbstractEvaluator):20+ENDPOINT_PATH = "call/ai-activity-type"21+REQUEST_MODEL = AiActivityTypeRequest22+RESPONSE_MODEL = AiActivityTypeResponse23+OUTPUT_SUBFOLDER = "ai_activity_type"24+USES_LLM_EVALUATION = False25+26+READS_FROM_DATASET_FOLDER = True27+28+DATA_SET: str = "initial"29+30+MINIMUM_REQUEST_BODY_SIZE: int = 031+32+SKIP_UNREPLAYABLE_ON_COLLECT: bool = True33+REPLAYABILITY_CHECK_TIMEOUT_SECONDS: float = 20.034+35+POPULATE_GROUND_TRUTH_LLM: bool = False36+GROUND_TRUTH_LLM_ENDPOINT: str = "OPENAI:gpt-5_2"37+38+COMPARE_TO_GROUND_TRUTH: str = "es"39+40+def __init__(self, **kwargs: Any) -> None:41+super().__init__(**kwargs)42+_repo_root = Path(__file__).parents[2]43+_datasets_root = _repo_root / "eval_output" / self.OUTPUT_SUBFOLDER / "eval_datasets"44+_dataset_dir = _datasets_root / self.__class__.DATA_SET45+self._dataset_dir = _dataset_dir46+self._inputs_dir = _dataset_dir / "input"47+self._ground_truths_dir = _dataset_dir / "ground_truth"48+49+async def collect_test_set(self) -> List[Dict[str, Any]]:50+if self.es_host_name:51+return await self._collect_from_es()52+return await self._collect_from_files()53+54+async def _collect_from_es(self) -> List[Dict[str, Any]]:55+if self._inputs_dir.exists() or self._ground_truths_dir.exists():56+raise FileExistsError(57+f"Dataset folder already exists: {self._inputs_dir.parent}. "58+"Choose a different --data-set name or remove the folder manually."59+ )60+61+raw_samples = await es_collect_test_set(62+es_host_name=self.es_host_name,63+es_port=self.es_port,64+endpoint_path=self.ENDPOINT_PATH,65+limit=self.limit,66+language_filter=self.language_filter,67+min_prompt_length=self.min_prompt_length,68+is_json_response=self.IS_JSON_RESPONSE,69+exclude_languages=self.exclude_languages,70+ )71+72+self._inputs_dir.mkdir(parents=True, exist_ok=True)73+self._ground_truths_dir.mkdir(parents=True, exist_ok=True)74+75+min_size = self.MINIMUM_REQUEST_BODY_SIZE76+samples: List[Dict[str, Any]] = []77+skipped_no_response = 078+skipped_too_small = 079+skipped_unreplayable = 080+skipped_probe_errors = 081+82+endpoint_url = f"{self.api_host}/{self.ENDPOINT_PATH}"83+84+timeout = aiohttp.ClientTimeout(total=self.REPLAYABILITY_CHECK_TIMEOUT_SECONDS)85+async with aiohttp.ClientSession(timeout=timeout) as session:86+for sample in raw_samples:87+es_id = sample["id"]88+request_body = sample["request_body"]89+original_response = sample.get("original_response")90+91+if not original_response:92+skipped_no_response += 193+continue94+95+if min_size and len(json.dumps(request_body)) < min_size:96+skipped_too_small += 197+continue98+99+if self.SKIP_UNREPLAYABLE_ON_COLLECT:100+try:101+async with session.post(endpoint_url, json=request_body) as resp:102+if resp.status != 200:103+body_text = await resp.text()104+if "Cannot find activity" in body_text:105+skipped_unreplayable += 1106+continue107+except Exception:108+skipped_probe_errors += 1109+110+with open(self._inputs_dir / f"{es_id}.json", "w") as f:111+json.dump(request_body, f, indent=2)112+113+with open(self._ground_truths_dir / f"gt_{es_id}.json", "w") as f:114+json.dump(original_response, f, indent=2)115+116+samples.append(117+ {118+"id": es_id,119+"request_body": request_body,120+"original_response": original_response,121+"ground_truth": original_response,122+ }123+ )124+125+print(f"Wrote {len(samples)} samples to {self._inputs_dir.parent}")126+if skipped_no_response:127+print(f"Skipped {skipped_no_response} records with no response body")128+if skipped_too_small:129+print(130+f"Skipped {skipped_too_small} records whose request body was smaller "131+f"than {min_size} characters"132+ )133+if skipped_unreplayable:134+print(f"Skipped {skipped_unreplayable} records that are not replayable (missing activity in ES)")135+if skipped_probe_errors:136+print(f"Warning: replayability probe errored for {skipped_probe_errors} records (kept those records)")137+138+if self.__class__.POPULATE_GROUND_TRUTH_LLM:139+label_results = await run_endpoint_http(140+test_set=samples,141+endpoint_url=endpoint_url,142+request_model=self.REQUEST_MODEL,143+force_llm_endpoint=self.__class__.GROUND_TRUTH_LLM_ENDPOINT,144+is_streaming_response=self.IS_STREAMING_RESPONSE,145+timeout_seconds=300.0,146+parallel_requests=self.parallel_requests,147+return_prompt_in_run=self.RETURN_PROMPT_IN_RUN,148+ )149+150+for i, r in enumerate(label_results):151+forced = self.__class__.GROUND_TRUTH_LLM_ENDPOINT152+model = forced.split(":", 1)[1] if ":" in forced else forced153+samples[i]["ground_truth_llm_model"] = model154+155+if not r.get("success", True):156+samples[i]["ground_truth_llm"] = None157+samples[i]["ground_truth_llm_reasoning"] = None158+samples[i]["ground_truth_llm_error_type"] = r.get("error_type")159+samples[i]["ground_truth_llm_error_message"] = r.get("error_message")160+continue161+162+resp = r.get("new_response") or {}163+samples[i]["ground_truth_llm"] = resp.get("ai_activity_type")164+samples[i]["ground_truth_llm_reasoning"] = resp.get("ai_activity_type_reasoning")165+166+return samples167+168+async def _collect_from_files(self) -> List[Dict[str, Any]]:169+if not self._inputs_dir.exists():170+raise FileNotFoundError(171+f"Dataset folder not found: {self._inputs_dir}. "172+"Run --steps collect first or check your --data-set name."173+ )174+175+samples: List[Dict[str, Any]] = []176+for filename in sorted(os.listdir(self._inputs_dir)):177+input_path = self._inputs_dir / filename178+gt_path = self._ground_truths_dir / f"gt_{filename}"179+180+if not gt_path.exists():181+print(f"Warning: ground truth not found for {filename}, skipping")182+continue183+184+with open(input_path, "r") as f:185+request_body: Dict[str, Any] = json.load(f)186+187+with open(gt_path, "r") as f:188+ground_truth: Dict[str, Any] = json.load(f)189+190+samples.append(191+ {192+"id": Path(filename).stem,193+"request_body": request_body,194+"original_response": None,195+"ground_truth": ground_truth,196+ }197+ )198+199+print(f"Loaded {len(samples)} samples from {self._inputs_dir.parent}")200+return samples201+202+async def evaluate_against_ground_truth(self, results: List[Dict[str, Any]]) -> List[EvaluationResult]:203+evaluation_results: List[EvaluationResult] = []204+compare_to = self.__class__.COMPARE_TO_GROUND_TRUTH205+206+for res in results:207+req_id = res["request_id"]208+is_success = res.get("success", True)209+210+if not is_success:211+evaluation_results.append(212+EvaluationResult(213+request_id=req_id,214+request_body=res["request_body"],215+new_response=res.get("new_response"),216+valid_schema=False,217+quality_score=0.0,218+quality_feedback=f"Request failed: {res.get('error_type')} - {res.get('error_message')}",219+response_time=res.get("response_time"),220+success=False,221+error_type=res.get("error_type"),222+error_message=res.get("error_message"),223+ground_truth_match=False,224+ )225+ )226+continue227+228+pred = (res.get("new_response") or {}).get("ai_activity_type")229+230+if compare_to == "llm":231+if "ground_truth_llm" not in res:232+evaluation_results.append(233+EvaluationResult(234+request_id=req_id,235+request_body=res["request_body"],236+new_response=res.get("new_response"),237+valid_schema=res.get("valid_schema", True),238+quality_score=0.0,239+quality_feedback=(240+"Missing ground_truth_llm in run results. Ensure collect step populated it and "241+"endpoints.py passes it through."242+ ),243+response_time=res.get("response_time"),244+success=True,245+ground_truth_match=False,246+ )247+ )248+continue249+gt_label = res.get("ground_truth_llm")250+else:251+gt_path = self._ground_truths_dir / f"gt_{req_id}.json"252+if not gt_path.exists():253+evaluation_results.append(254+EvaluationResult(255+request_id=req_id,256+request_body=res["request_body"],257+new_response=res.get("new_response"),258+valid_schema=res.get("valid_schema", True),259+quality_score=0.0,260+quality_feedback=f"Ground truth file not found: {gt_path}",261+response_time=res.get("response_time"),262+success=True,263+ground_truth_match=False,264+ )265+ )266+continue267+268+with open(gt_path, "r") as f:269+ground_truth_dict: Dict[str, Any] = json.load(f)270+271+gt_label = (ground_truth_dict or {}).get("ai_activity_type")272+273+matched = pred == gt_label274+275+evaluation_results.append(276+EvaluationResult(277+request_id=req_id,278+request_body=res["request_body"],279+new_response=res.get("new_response"),280+valid_schema=res.get("valid_schema", True),281+quality_score=1.0 if matched else 0.0,282+quality_feedback="PASS" if matched else f"FAIL\nExpected: {gt_label}\nGot: {pred}",283+response_time=res.get("response_time"),284+success=True,285+ground_truth_match=matched,286+ )287+ )288+289+return evaluation_results290+291+292+if __name__ == "__main__":293+parser = create_parser(AiActivityTypeEvaluator)294+parser.add_argument(295+"--data-set",296+type=str,297+default="initial",298+dest="data_set",299+help="Dataset folder under eval_output/ai_activity_type/eval_datasets/ (default: initial)",300+ )301+302+parser.add_argument(303+"--populate-ground-truth-llm",304+action="store_true",305+dest="populate_ground_truth_llm",306+ )307+parser.add_argument(308+"--ground-truth-llm-endpoint",309+type=str,310+default="OPENAI:gpt-5_2",311+dest="ground_truth_llm_endpoint",312+ )313+314+parser.add_argument(315+"--compare-to-ground-truth",316+type=str,317+choices=["es", "llm"],318+default="es",319+dest="compare_to_ground_truth",320+ )321+322+args = parser.parse_args()323+AiActivityTypeEvaluator.DATA_SET = args.data_set324+325+AiActivityTypeEvaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_llm)326+AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpoint327+328+AiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truth329+330+asyncio.run(run_evaluation(AiActivityTypeEvaluator, args))</selection>” selected. Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
You said
I’m on page “<tabTitle>Jy 19798 evaluation for ai activity types by nikol</tabTitle>” with “<selection>@@ -0,0 +1,330 @@1+#!/usr/bin/env python32+import asyncio3+import json4+import os5+from pathlib import Path6+from typing import Any, Dict, List7+8+import aiohttp9+10+from src.component.call_processing.schemes.ai_activity_type_request import AiActivityTypeRequest11+from src.component.call_processing.schemes.ai_activity_type_response import AiActivityTypeResponse12+from src.eval.abstract_evaluator import AbstractEvaluator13+from src.eval.collectors import collect_test_set as es_collect_test_set14+from src.eval.common_evaluator_runner import create_parser, run_evaluation15+from src.eval.endpoints import run_endpoint as run_endpoint_http16+from src.eval.models import EvaluationResult17+18+19+class AiActivityTypeEvaluator(AbstractEvaluator):20+ENDPOINT_PATH = "call/ai-activity-type"21+REQUEST_MODEL = AiActivityTypeRequest22+RESPONSE_MODEL = AiActivityTypeResponse23+OUTPUT_SUBFOLDER = "ai_activity_type"24+USES_LLM_EVALUATION = False25+26+READS_FROM_DATASET_FOLDER = True27+28+DATA_SET: str = "initial"29+30+MINIMUM_REQUEST_BODY_SIZE: int = 031+32+SKIP_UNREPLAYABLE_ON_COLLECT: bool = True33+REPLAYABILITY_CHECK_TIMEOUT_SECONDS: float = 20.034+35+POPULATE_GROUND_TRUTH_LLM: bool = False36+GROUND_TRUTH_LLM_ENDPOINT: str = "OPENAI:gpt-5_2"37+38+COMPARE_TO_GROUND_TRUTH: str = "es"39+40+def __init__(self, **kwargs: Any) -> None:41+super().__init__(**kwargs)42+_repo_root = Path(__file__).parents[2]43+_datasets_root = _repo_root / "eval_output" / self.OUTPUT_SUBFOLDER / "eval_datasets"44+_dataset_dir = _datasets_root / self.__class__.DATA_SET45+self._dataset_dir = _dataset_dir46+self._inputs_dir = _dataset_dir / "input"47+self._ground_truths_dir = _dataset_dir / "ground_truth"48+49+async def collect_test_set(self) -> List[Dict[str, Any]]:50+if self.es_host_name:51+return await self._collect_from_es()52+return await self._collect_from_files()53+54+async def _collect_from_es(self) -> List[Dict[str, Any]]:55+if self._inputs_dir.exists() or self._ground_truths_dir.exists():56+raise FileExistsError(57+f"Dataset folder already exists: {self._inputs_dir.parent}. "58+"Choose a different --data-set name or remove the folder manually."59+ )60+61+raw_samples = await es_collect_test_set(62+es_host_name=self.es_host_name,63+es_port=self.es_port,64+endpoint_path=self.ENDPOINT_PATH,65+limit=self.limit,66+language_filter=self.language_filter,67+min_prompt_length=self.min_prompt_length,68+is_json_response=self.IS_JSON_RESPONSE,69+exclude_languages=self.exclude_languages,70+ )71+72+self._inputs_dir.mkdir(parents=True, exist_ok=True)73+self._ground_truths_dir.mkdir(parents=True, exist_ok=True)74+75+min_size = self.MINIMUM_REQUEST_BODY_SIZE76+samples: List[Dict[str, Any]] = []77+skipped_no_response = 078+skipped_too_small = 079+skipped_unreplayable = 080+skipped_probe_errors = 081+82+endpoint_url = f"{self.api_host}/{self.ENDPOINT_PATH}"83+84+timeout = aiohttp.ClientTimeout(total=self.REPLAYABILITY_CHECK_TIMEOUT_SECONDS)85+async with aiohttp.ClientSession(timeout=timeout) as session:86+for sample in raw_samples:87+es_id = sample["id"]88+request_body = sample["request_body"]89+original_response = sample.get("original_response")90+91+if not original_response:92+skipped_no_response += 193+continue94+95+if min_size and len(json.dumps(request_body)) < min_size:96+skipped_too_small += 197+continue98+99+if self.SKIP_UNREPLAYABLE_ON_COLLECT:100+try:101+async with session.post(endpoint_url, json=request_body) as resp:102+if resp.status != 200:103+body_text = await resp.text()104+if "Cannot find activity" in body_text:105+skipped_unreplayable += 1106+continue107+except Exception:108+skipped_probe_errors += 1109+110+with open(self._inputs_dir / f"{es_id}.json", "w") as f:111+json.dump(request_body, f, indent=2)112+113+with open(self._ground_truths_dir / f"gt_{es_id}.json", "w") as f:114+json.dump(original_response, f, indent=2)115+116+samples.append(117+ {118+"id": es_id,119+"request_body": request_body,120+"original_response": original_response,121+"ground_truth": original_response,122+ }123+ )124+125+print(f"Wrote {len(samples)} samples to {self._inputs_dir.parent}")126+if skipped_no_response:127+print(f"Skipped {skipped_no_response} records with no response body")128+if skipped_too_small:129+print(130+f"Skipped {skipped_too_small} records whose request body was smaller "131+f"than {min_size} characters"132+ )133+if skipped_unreplayable:134+print(f"Skipped {skipped_unreplayable} records that are not replayable (missing activity in ES)")135+if skipped_probe_errors:136+print(f"Warning: replayability probe errored for {skipped_probe_errors} records (kept those records)")137+138+if self.__class__.POPULATE_GROUND_TRUTH_LLM:139+label_results = await run_endpoint_http(140+test_set=samples,141+endpoint_url=endpoint_url,142+request_model=self.REQUEST_MODEL,143+force_llm_endpoint=self.__class__.GROUND_TRUTH_LLM_ENDPOINT,144+is_streaming_response=self.IS_STREAMING_RESPONSE,145+timeout_seconds=300.0,146+parallel_requests=self.parallel_requests,147+return_prompt_in_run=self.RETURN_PROMPT_IN_RUN,148+ )149+150+for i, r in enumerate(label_results):151+forced = self.__class__.GROUND_TRUTH_LLM_ENDPOINT152+model = forced.split(":", 1)[1] if ":" in forced else forced153+samples[i]["ground_truth_llm_model"] = model154+155+if not r.get("success", True):156+samples[i]["ground_truth_llm"] = None157+samples[i]["ground_truth_llm_reasoning"] = None158+samples[i]["ground_truth_llm_error_type"] = r.get("error_type")159+samples[i]["ground_truth_llm_error_message"] = r.get("error_message")160+continue161+162+resp = r.get("new_response") or {}163+samples[i]["ground_truth_llm"] = resp.get("ai_activity_type")164+samples[i]["ground_truth_llm_reasoning"] = resp.get("ai_activity_type_reasoning")165+166+return samples167+168+async def _collect_from_files(self) -> List[Dict[str, Any]]:169+if not self._inputs_dir.exists():170+raise FileNotFoundError(171+f"Dataset folder not found: {self._inputs_dir}. "172+"Run --steps collect first or check your --data-set name."173+ )174+175+samples: List[Dict[str, Any]] = []176+for filename in sorted(os.listdir(self._inputs_dir)):177+input_path = self._inputs_dir / filename178+gt_path = self._ground_truths_dir / f"gt_{filename}"179+180+if not gt_path.exists():181+print(f"Warning: ground truth not found for {filename}, skipping")182+continue183+184+with open(input_path, "r") as f:185+request_body: Dict[str, Any] = json.load(f)186+187+with open(gt_path, "r") as f:188+ground_truth: Dict[str, Any] = json.load(f)189+190+samples.append(191+ {192+"id": Path(filename).stem,193+"request_body": request_body,194+"original_response": None,195+"ground_truth": ground_truth,196+ }197+ )198+199+print(f"Loaded {len(samples)} samples from {self._inputs_dir.parent}")200+return samples201+202+async def evaluate_against_ground_truth(self, results: List[Dict[str, Any]]) -> List[EvaluationResult]:203+evaluation_results: List[EvaluationResult] = []204+compare_to = self.__class__.COMPARE_TO_GROUND_TRUTH205+206+for res in results:207+req_id = res["request_id"]208+is_success = res.get("success", True)209+210+if not is_success:211+evaluation_results.append(212+EvaluationResult(213+request_id=req_id,214+request_body=res["request_body"],215+new_response=res.get("new_response"),216+valid_schema=False,217+quality_score=0.0,218+quality_feedback=f"Request failed: {res.get('error_type')} - {res.get('error_message')}",219+response_time=res.get("response_time"),220+success=False,221+error_type=res.get("error_type"),222+error_message=res.get("error_message"),223+ground_truth_match=False,224+ )225+ )226+continue227+228+pred = (res.get("new_response") or {}).get("ai_activity_type")229+230+if compare_to == "llm":231+if "ground_truth_llm" not in res:232+evaluation_results.append(233+EvaluationResult(234+request_id=req_id,235+request_body=res["request_body"],236+new_response=res.get("new_response"),237+valid_schema=res.get("valid_schema", True),238+quality_score=0.0,239+quality_feedback=(240+"Missing ground_truth_llm in run results. Ensure collect step populated it and "241+"endpoints.py passes it through."242+ ),243+response_time=res.get("response_time"),244+success=True,245+ground_truth_match=False,246+ )247+ )248+continue249+gt_label = res.get("ground_truth_llm")250+else:251+gt_path = self._ground_truths_dir / f"gt_{req_id}.json"252+if not gt_path.exists():253+evaluation_results.append(254+EvaluationResult(255+request_id=req_id,256+request_body=res["request_body"],257+new_response=res.get("new_response"),258+valid_schema=res.get("valid_schema", True),259+quality_score=0.0,260+quality_feedback=f"Ground truth file not found: {gt_path}",261+response_time=res.get("response_time"),262+success=True,263+ground_truth_match=False,264+ )265+ )266+continue267+268+with open(gt_path, "r") as f:269+ground_truth_dict: Dict[str, Any] = json.load(f)270+271+gt_label = (ground_truth_dict or {}).get("ai_activity_type")272+273+matched = pred == gt_label274+275+evaluation_results.append(276+EvaluationResult(277+request_id=req_id,278+request_body=res["request_body"],279+new_response=res.get("new_response"),280+valid_schema=res.get("valid_schema", True),281+quality_score=1.0 if matched else 0.0,282+quality_feedback="PASS" if matched else f"FAIL\nExpected: {gt_label}\nGot: {pred}",283+response_time=res.get("response_time"),284+success=True,285+ground_truth_match=matched,286+ )287+ )288+289+return evaluation_results290+291+292+if __name__ == "__main__":293+parser = create_parser(AiActivityTypeEvaluator)294+parser.add_argument(295+"--data-set",296+type=str,297+default="initial",298+dest="data_set",299+help="Dataset folder under eval_output/ai_activity_type/eval_datasets/ (default: initial)",300+ )301+302+parser.add_argument(303+"--populate-ground-truth-llm",304+action="store_true",305+dest="populate_ground_truth_llm",306+ )307+parser.add_argument(308+"--ground-truth-llm-endpoint",309+type=str,310+default="OPENAI:gpt-5_2",311+dest="ground_truth_llm_endpoint",312+ )313+314+parser.add_argument(315+"--compare-to-ground-truth",316+type=str,317+choices=["es", "llm"],318+default="es",319+dest="compare_to_ground_truth",320+ )321+322+args = parser.parse_args()323+AiActivityTypeEvaluator.DATA_SET = args.data_set324+325+AiActivityTypeEvaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_llm)326+AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpoint327+328+AiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truth329+330+asyncio.run(run_evaluation(AiActivityTypeEvaluator, args))</selection>” selected.
Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
Expand
Listen
Show more options
Gemini said
Gemini said
This Python script defines the
AiActivityTypeEvaluator
class, a specialized tool for evaluating AI-driven activity type classification. It extends a base evaluation framework to handle data collection, ground-truth comparison, and performance scoring.
Core Functionality
Core Functionality
Data Collection (
collect_test_set
):
* Supports fetching raw samples from
Elasticsearch (ES)
or loading them from
local JSON files
.
Implements a
replayability check
: It probes the live API endpoint to ensure the activity exists before including it in the dataset.
Filters data based on request body size, language, and the presence of an original response.
Ground Truth Management:
Saves inputs and "gold standard" responses to a local directory structure (
eval_output/ai_activity_type/...
).
Features an optional mode to populate ground truth using a high-order LLM (e.g.,
gpt-5_2
) if the
POPULATE_GROUND_TRUTH_LLM
flag is set.
Evaluation Logic (
evaluate_against_ground_truth
):
Compares the model's predicted
ai_activity_type
against either the original ES response or the LLM-generated ground truth.
Assigns a
quality_score
of
1.0
for a match and
0.0
for a mismatch or request failure.
Key Configuration Constants
Key Configuration Constants
Constant
Purpose
ENDPOINT_PATH
The specific API route being tested:
call/ai-activity-type
.
COMPARE_TO_GROUND_TRUTH
Determines if the baseline is the original data (
es
) or an
llm
.
SKIP_UNREPLAYABLE_ON_COLLECT
If
True
, discards records where the API returns "Cannot find activity."
REPLAYABILITY_CHECK_TIMEOUT_SECONDS
Limits the probe request to 20 seconds.
Constant
ENDPOINT_PATH
COMPARE_TO_GROUND_TRUTH
SKIP_UNREPLAYABLE_ON_COLLECT
REPLAYABILITY_CHECK_TIMEOUT_SECONDS
Purpose
The specific API route being tested:
call/ai-activity-type
.
Determines if the baseline is the original data (
es
) or an
llm
.
If
True
, discards records where the API returns "Cannot find activity."
Limits the probe request to 20 seconds.
Copy table
Execution Workflow
Execution Workflow
Parsing:
The script uses
create_parser
to handle CLI arguments like
--data-set
,
--populate-ground-truth-llm
, and
--compare-to-ground-truth
.
Initialization:
Sets up directory paths for inputs and ground truths based on the selected dataset name.
Run:
Executes the evaluation via
asyncio.run(run_evaluation(...))
, which orchestrates the collection, endpoint calling, and result scoring phases.
Copy prompt
Edit
You said are there soem issues with this PR
You said
are there soem issues with this PR
Show more options
Analyzing Implementation
Gemini said
Gemini said
Enter a prompt for Gemini
encrypted
Enter a prompt for Gemini
encrypted
Open upload file menu
Tools
Open mode picker
Pro
Stop response
Your Jiminny chats aren’t used to improve our models. Gemini is AI and can make mistakes, including about people.
Your privacy & Gemini Opens in a new window
Your privacy & Gemini
Opens in a new window
Summarize page
Summarize page
Skip to content
Skip to content
Open menu
Homepage (g then d)
jiminny
jiminny
prophet
prophet
Search or jump to…
Type
/
to search
Chat with Copilot
Open Copilot…
Create new...
Issues
Pull requests
Repositories
You have unread notifications(g then n)
Open user navigation menu
Repository navigation
Repository navigation
Code
Code
Issues
Issues
Pull requests (6)
Pull requests
(
6
)
Agents...
|
Firefox
|
Jy 19798 evaluation for ai activity types by nikol Jy 19798 evaluation for ai activity types by nikolaybiaivanov · Pull Request #468 · jiminny/prophet — Work...
|
github.com/jiminny/prophet/pull/468/changes#diff-d github.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8...
|
8542
|
|
JY-20543 add AJ reports User pilot tracking by Lak JY-20543 add AJ reports User pilot tracking by LakyLak · Pull Request #11932 · jiminny/app
Platform Sprint 1 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 1 Q2 - Platform Team - Scrum Board - Jira
Jy 19798 evaluation for ai activity types by nikolaybiaivanov · Pull Request #468 · jiminny/prophet
Jy 19798 evaluation for ai activity types by nikolaybiaivanov · Pull Request #468 · jiminny/prophet
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Copy prompt
You said I’m on page “<tabTitle>Jy 19798 evaluation for ai activity types by nikol</tabTitle>” with “<selection>@@ -0,0 +1,330 @@1+#!/usr/bin/env python32+import asyncio3+import json4+import os5+from pathlib import Path6+from typing import Any, Dict, List7+8+import aiohttp9+10+from src.component.call_processing.schemes.ai_activity_type_request import AiActivityTypeRequest11+from src.component.call_processing.schemes.ai_activity_type_response import AiActivityTypeResponse12+from src.eval.abstract_evaluator import AbstractEvaluator13+from src.eval.collectors import collect_test_set as es_collect_test_set14+from src.eval.common_evaluator_runner import create_parser, run_evaluation15+from src.eval.endpoints import run_endpoint as run_endpoint_http16+from src.eval.models import EvaluationResult17+18+19+class AiActivityTypeEvaluator(AbstractEvaluator):20+ENDPOINT_PATH = "call/ai-activity-type"21+REQUEST_MODEL = AiActivityTypeRequest22+RESPONSE_MODEL = AiActivityTypeResponse23+OUTPUT_SUBFOLDER = "ai_activity_type"24+USES_LLM_EVALUATION = False25+26+READS_FROM_DATASET_FOLDER = True27+28+DATA_SET: str = "initial"29+30+MINIMUM_REQUEST_BODY_SIZE: int = 031+32+SKIP_UNREPLAYABLE_ON_COLLECT: bool = True33+REPLAYABILITY_CHECK_TIMEOUT_SECONDS: float = 20.034+35+POPULATE_GROUND_TRUTH_LLM: bool = False36+GROUND_TRUTH_LLM_ENDPOINT: str = "OPENAI:gpt-5_2"37+38+COMPARE_TO_GROUND_TRUTH: str = "es"39+40+def __init__(self, **kwargs: Any) -> None:41+super().__init__(**kwargs)42+_repo_root = Path(__file__).parents[2]43+_datasets_root = _repo_root / "eval_output" / self.OUTPUT_SUBFOLDER / "eval_datasets"44+_dataset_dir = _datasets_root / self.__class__.DATA_SET45+self._dataset_dir = _dataset_dir46+self._inputs_dir = _dataset_dir / "input"47+self._ground_truths_dir = _dataset_dir / "ground_truth"48+49+async def collect_test_set(self) -> List[Dict[str, Any]]:50+if self.es_host_name:51+return await self._collect_from_es()52+return await self._collect_from_files()53+54+async def _collect_from_es(self) -> List[Dict[str, Any]]:55+if self._inputs_dir.exists() or self._ground_truths_dir.exists():56+raise FileExistsError(57+f"Dataset folder already exists: {self._inputs_dir.parent}. "58+"Choose a different --data-set name or remove the folder manually."59+ )60+61+raw_samples = await es_collect_test_set(62+es_host_name=self.es_host_name,63+es_port=self.es_port,64+endpoint_path=self.ENDPOINT_PATH,65+limit=self.limit,66+language_filter=self.language_filter,67+min_prompt_length=self.min_prompt_length,68+is_json_response=self.IS_JSON_RESPONSE,69+exclude_languages=self.exclude_languages,70+ )71+72+self._inputs_dir.mkdir(parents=True, exist_ok=True)73+self._ground_truths_dir.mkdir(parents=True, exist_ok=True)74+75+min_size = self.MINIMUM_REQUEST_BODY_SIZE76+samples: List[Dict[str, Any]] = []77+skipped_no_response = 078+skipped_too_small = 079+skipped_unreplayable = 080+skipped_probe_errors = 081+82+endpoint_url = f"{self.api_host}/{self.ENDPOINT_PATH}"83+84+timeout = aiohttp.ClientTimeout(total=self.REPLAYABILITY_CHECK_TIMEOUT_SECONDS)85+async with aiohttp.ClientSession(timeout=timeout) as session:86+for sample in raw_samples:87+es_id = sample["id"]88+request_body = sample["request_body"]89+original_response = sample.get("original_response")90+91+if not original_response:92+skipped_no_response += 193+continue94+95+if min_size and len(json.dumps(request_body)) < min_size:96+skipped_too_small += 197+continue98+99+if self.SKIP_UNREPLAYABLE_ON_COLLECT:100+try:101+async with session.post(endpoint_url, json=request_body) as resp:102+if resp.status != 200:103+body_text = await resp.text()104+if "Cannot find activity" in body_text:105+skipped_unreplayable += 1106+continue107+except Exception:108+skipped_probe_errors += 1109+110+with open(self._inputs_dir / f"{es_id}.json", "w") as f:111+json.dump(request_body, f, indent=2)112+113+with open(self._ground_truths_dir / f"gt_{es_id}.json", "w") as f:114+json.dump(original_response, f, indent=2)115+116+samples.append(117+ {118+"id": es_id,119+"request_body": request_body,120+"original_response": original_response,121+"ground_truth": original_response,122+ }123+ )124+125+print(f"Wrote {len(samples)} samples to {self._inputs_dir.parent}")126+if skipped_no_response:127+print(f"Skipped {skipped_no_response} records with no response body")128+if skipped_too_small:129+print(130+f"Skipped {skipped_too_small} records whose request body was smaller "131+f"than {min_size} characters"132+ )133+if skipped_unreplayable:134+print(f"Skipped {skipped_unreplayable} records that are not replayable (missing activity in ES)")135+if skipped_probe_errors:136+print(f"Warning: replayability probe errored for {skipped_probe_errors} records (kept those records)")137+138+if self.__class__.POPULATE_GROUND_TRUTH_LLM:139+label_results = await run_endpoint_http(140+test_set=samples,141+endpoint_url=endpoint_url,142+request_model=self.REQUEST_MODEL,143+force_llm_endpoint=self.__class__.GROUND_TRUTH_LLM_ENDPOINT,144+is_streaming_response=self.IS_STREAMING_RESPONSE,145+timeout_seconds=300.0,146+parallel_requests=self.parallel_requests,147+return_prompt_in_run=self.RETURN_PROMPT_IN_RUN,148+ )149+150+for i, r in enumerate(label_results):151+forced = self.__class__.GROUND_TRUTH_LLM_ENDPOINT152+model = forced.split(":", 1)[1] if ":" in forced else forced153+samples[i]["ground_truth_llm_model"] = model154+155+if not r.get("success", True):156+samples[i]["ground_truth_llm"] = None157+samples[i]["ground_truth_llm_reasoning"] = None158+samples[i]["ground_truth_llm_error_type"] = r.get("error_type")159+samples[i]["ground_truth_llm_error_message"] = r.get("error_message")160+continue161+162+resp = r.get("new_response") or {}163+samples[i]["ground_truth_llm"] = resp.get("ai_activity_type")164+samples[i]["ground_truth_llm_reasoning"] = resp.get("ai_activity_type_reasoning")165+166+return samples167+168+async def _collect_from_files(self) -> List[Dict[str, Any]]:169+if not self._inputs_dir.exists():170+raise FileNotFoundError(171+f"Dataset folder not found: {self._inputs_dir}. "172+"Run --steps collect first or check your --data-set name."173+ )174+175+samples: List[Dict[str, Any]] = []176+for filename in sorted(os.listdir(self._inputs_dir)):177+input_path = self._inputs_dir / filename178+gt_path = self._ground_truths_dir / f"gt_{filename}"179+180+if not gt_path.exists():181+print(f"Warning: ground truth not found for {filename}, skipping")182+continue183+184+with open(input_path, "r") as f:185+request_body: Dict[str, Any] = json.load(f)186+187+with open(gt_path, "r") as f:188+ground_truth: Dict[str, Any] = json.load(f)189+190+samples.append(191+ {192+"id": Path(filename).stem,193+"request_body": request_body,194+"original_response": None,195+"ground_truth": ground_truth,196+ }197+ )198+199+print(f"Loaded {len(samples)} samples from {self._inputs_dir.parent}")200+return samples201+202+async def evaluate_against_ground_truth(self, results: List[Dict[str, Any]]) -> List[EvaluationResult]:203+evaluation_results: List[EvaluationResult] = []204+compare_to = self.__class__.COMPARE_TO_GROUND_TRUTH205+206+for res in results:207+req_id = res["request_id"]208+is_success = res.get("success", True)209+210+if not is_success:211+evaluation_results.append(212+EvaluationResult(213+request_id=req_id,214+request_body=res["request_body"],215+new_response=res.get("new_response"),216+valid_schema=False,217+quality_score=0.0,218+quality_feedback=f"Request failed: {res.get('error_type')} - {res.get('error_message')}",219+response_time=res.get("response_time"),220+success=False,221+error_type=res.get("error_type"),222+error_message=res.get("error_message"),223+ground_truth_match=False,224+ )225+ )226+continue227+228+pred = (res.get("new_response") or {}).get("ai_activity_type")229+230+if compare_to == "llm":231+if "ground_truth_llm" not in res:232+evaluation_results.append(233+EvaluationResult(234+request_id=req_id,235+request_body=res["request_body"],236+new_response=res.get("new_response"),237+valid_schema=res.get("valid_schema", True),238+quality_score=0.0,239+quality_feedback=(240+"Missing ground_truth_llm in run results. Ensure collect step populated it and "241+"endpoints.py passes it through."242+ ),243+response_time=res.get("response_time"),244+success=True,245+ground_truth_match=False,246+ )247+ )248+continue249+gt_label = res.get("ground_truth_llm")250+else:251+gt_path = self._ground_truths_dir / f"gt_{req_id}.json"252+if not gt_path.exists():253+evaluation_results.append(254+EvaluationResult(255+request_id=req_id,256+request_body=res["request_body"],257+new_response=res.get("new_response"),258+valid_schema=res.get("valid_schema", True),259+quality_score=0.0,260+quality_feedback=f"Ground truth file not found: {gt_path}",261+response_time=res.get("response_time"),262+success=True,263+ground_truth_match=False,264+ )265+ )266+continue267+268+with open(gt_path, "r") as f:269+ground_truth_dict: Dict[str, Any] = json.load(f)270+271+gt_label = (ground_truth_dict or {}).get("ai_activity_type")272+273+matched = pred == gt_label274+275+evaluation_results.append(276+EvaluationResult(277+request_id=req_id,278+request_body=res["request_body"],279+new_response=res.get("new_response"),280+valid_schema=res.get("valid_schema", True),281+quality_score=1.0 if matched else 0.0,282+quality_feedback="PASS" if matched else f"FAIL\nExpected: {gt_label}\nGot: {pred}",283+response_time=res.get("response_time"),284+success=True,285+ground_truth_match=matched,286+ )287+ )288+289+return evaluation_results290+291+292+if __name__ == "__main__":293+parser = create_parser(AiActivityTypeEvaluator)294+parser.add_argument(295+"--data-set",296+type=str,297+default="initial",298+dest="data_set",299+help="Dataset folder under eval_output/ai_activity_type/eval_datasets/ (default: initial)",300+ )301+302+parser.add_argument(303+"--populate-ground-truth-llm",304+action="store_true",305+dest="populate_ground_truth_llm",306+ )307+parser.add_argument(308+"--ground-truth-llm-endpoint",309+type=str,310+default="OPENAI:gpt-5_2",311+dest="ground_truth_llm_endpoint",312+ )313+314+parser.add_argument(315+"--compare-to-ground-truth",316+type=str,317+choices=["es", "llm"],318+default="es",319+dest="compare_to_ground_truth",320+ )321+322+args = parser.parse_args()323+AiActivityTypeEvaluator.DATA_SET = args.data_set324+325+AiActivityTypeEvaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_llm)326+AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpoint327+328+AiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truth329+330+asyncio.run(run_evaluation(AiActivityTypeEvaluator, args))</selection>” selected. Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
You said
I’m on page “<tabTitle>Jy 19798 evaluation for ai activity types by nikol</tabTitle>” with “<selection>@@ -0,0 +1,330 @@1+#!/usr/bin/env python32+import asyncio3+import json4+import os5+from pathlib import Path6+from typing import Any, Dict, List7+8+import aiohttp9+10+from src.component.call_processing.schemes.ai_activity_type_request import AiActivityTypeRequest11+from src.component.call_processing.schemes.ai_activity_type_response import AiActivityTypeResponse12+from src.eval.abstract_evaluator import AbstractEvaluator13+from src.eval.collectors import collect_test_set as es_collect_test_set14+from src.eval.common_evaluator_runner import create_parser, run_evaluation15+from src.eval.endpoints import run_endpoint as run_endpoint_http16+from src.eval.models import EvaluationResult17+18+19+class AiActivityTypeEvaluator(AbstractEvaluator):20+ENDPOINT_PATH = "call/ai-activity-type"21+REQUEST_MODEL = AiActivityTypeRequest22+RESPONSE_MODEL = AiActivityTypeResponse23+OUTPUT_SUBFOLDER = "ai_activity_type"24+USES_LLM_EVALUATION = False25+26+READS_FROM_DATASET_FOLDER = True27+28+DATA_SET: str = "initial"29+30+MINIMUM_REQUEST_BODY_SIZE: int = 031+32+SKIP_UNREPLAYABLE_ON_COLLECT: bool = True33+REPLAYABILITY_CHECK_TIMEOUT_SECONDS: float = 20.034+35+POPULATE_GROUND_TRUTH_LLM: bool = False36+GROUND_TRUTH_LLM_ENDPOINT: str = "OPENAI:gpt-5_2"37+38+COMPARE_TO_GROUND_TRUTH: str = "es"39+40+def __init__(self, **kwargs: Any) -> None:41+super().__init__(**kwargs)42+_repo_root = Path(__file__).parents[2]43+_datasets_root = _repo_root / "eval_output" / self.OUTPUT_SUBFOLDER / "eval_datasets"44+_dataset_dir = _datasets_root / self.__class__.DATA_SET45+self._dataset_dir = _dataset_dir46+self._inputs_dir = _dataset_dir / "input"47+self._ground_truths_dir = _dataset_dir / "ground_truth"48+49+async def collect_test_set(self) -> List[Dict[str, Any]]:50+if self.es_host_name:51+return await self._collect_from_es()52+return await self._collect_from_files()53+54+async def _collect_from_es(self) -> List[Dict[str, Any]]:55+if self._inputs_dir.exists() or self._ground_truths_dir.exists():56+raise FileExistsError(57+f"Dataset folder already exists: {self._inputs_dir.parent}. "58+"Choose a different --data-set name or remove the folder manually."59+ )60+61+raw_samples = await es_collect_test_set(62+es_host_name=self.es_host_name,63+es_port=self.es_port,64+endpoint_path=self.ENDPOINT_PATH,65+limit=self.limit,66+language_filter=self.language_filter,67+min_prompt_length=self.min_prompt_length,68+is_json_response=self.IS_JSON_RESPONSE,69+exclude_languages=self.exclude_languages,70+ )71+72+self._inputs_dir.mkdir(parents=True, exist_ok=True)73+self._ground_truths_dir.mkdir(parents=True, exist_ok=True)74+75+min_size = self.MINIMUM_REQUEST_BODY_SIZE76+samples: List[Dict[str, Any]] = []77+skipped_no_response = 078+skipped_too_small = 079+skipped_unreplayable = 080+skipped_probe_errors = 081+82+endpoint_url = f"{self.api_host}/{self.ENDPOINT_PATH}"83+84+timeout = aiohttp.ClientTimeout(total=self.REPLAYABILITY_CHECK_TIMEOUT_SECONDS)85+async with aiohttp.ClientSession(timeout=timeout) as session:86+for sample in raw_samples:87+es_id = sample["id"]88+request_body = sample["request_body"]89+original_response = sample.get("original_response")90+91+if not original_response:92+skipped_no_response += 193+continue94+95+if min_size and len(json.dumps(request_body)) < min_size:96+skipped_too_small += 197+continue98+99+if self.SKIP_UNREPLAYABLE_ON_COLLECT:100+try:101+async with session.post(endpoint_url, json=request_body) as resp:102+if resp.status != 200:103+body_text = await resp.text()104+if "Cannot find activity" in body_text:105+skipped_unreplayable += 1106+continue107+except Exception:108+skipped_probe_errors += 1109+110+with open(self._inputs_dir / f"{es_id}.json", "w") as f:111+json.dump(request_body, f, indent=2)112+113+with open(self._ground_truths_dir / f"gt_{es_id}.json", "w") as f:114+json.dump(original_response, f, indent=2)115+116+samples.append(117+ {118+"id": es_id,119+"request_body": request_body,120+"original_response": original_response,121+"ground_truth": original_response,122+ }123+ )124+125+print(f"Wrote {len(samples)} samples to {self._inputs_dir.parent}")126+if skipped_no_response:127+print(f"Skipped {skipped_no_response} records with no response body")128+if skipped_too_small:129+print(130+f"Skipped {skipped_too_small} records whose request body was smaller "131+f"than {min_size} characters"132+ )133+if skipped_unreplayable:134+print(f"Skipped {skipped_unreplayable} records that are not replayable (missing activity in ES)")135+if skipped_probe_errors:136+print(f"Warning: replayability probe errored for {skipped_probe_errors} records (kept those records)")137+138+if self.__class__.POPULATE_GROUND_TRUTH_LLM:139+label_results = await run_endpoint_http(140+test_set=samples,141+endpoint_url=endpoint_url,142+request_model=self.REQUEST_MODEL,143+force_llm_endpoint=self.__class__.GROUND_TRUTH_LLM_ENDPOINT,144+is_streaming_response=self.IS_STREAMING_RESPONSE,145+timeout_seconds=300.0,146+parallel_requests=self.parallel_requests,147+return_prompt_in_run=self.RETURN_PROMPT_IN_RUN,148+ )149+150+for i, r in enumerate(label_results):151+forced = self.__class__.GROUND_TRUTH_LLM_ENDPOINT152+model = forced.split(":", 1)[1] if ":" in forced else forced153+samples[i]["ground_truth_llm_model"] = model154+155+if not r.get("success", True):156+samples[i]["ground_truth_llm"] = None157+samples[i]["ground_truth_llm_reasoning"] = None158+samples[i]["ground_truth_llm_error_type"] = r.get("error_type")159+samples[i]["ground_truth_llm_error_message"] = r.get("error_message")160+continue161+162+resp = r.get("new_response") or {}163+samples[i]["ground_truth_llm"] = resp.get("ai_activity_type")164+samples[i]["ground_truth_llm_reasoning"] = resp.get("ai_activity_type_reasoning")165+166+return samples167+168+async def _collect_from_files(self) -> List[Dict[str, Any]]:169+if not self._inputs_dir.exists():170+raise FileNotFoundError(171+f"Dataset folder not found: {self._inputs_dir}. "172+"Run --steps collect first or check your --data-set name."173+ )174+175+samples: List[Dict[str, Any]] = []176+for filename in sorted(os.listdir(self._inputs_dir)):177+input_path = self._inputs_dir / filename178+gt_path = self._ground_truths_dir / f"gt_{filename}"179+180+if not gt_path.exists():181+print(f"Warning: ground truth not found for {filename}, skipping")182+continue183+184+with open(input_path, "r") as f:185+request_body: Dict[str, Any] = json.load(f)186+187+with open(gt_path, "r") as f:188+ground_truth: Dict[str, Any] = json.load(f)189+190+samples.append(191+ {192+"id": Path(filename).stem,193+"request_body": request_body,194+"original_response": None,195+"ground_truth": ground_truth,196+ }197+ )198+199+print(f"Loaded {len(samples)} samples from {self._inputs_dir.parent}")200+return samples201+202+async def evaluate_against_ground_truth(self, results: List[Dict[str, Any]]) -> List[EvaluationResult]:203+evaluation_results: List[EvaluationResult] = []204+compare_to = self.__class__.COMPARE_TO_GROUND_TRUTH205+206+for res in results:207+req_id = res["request_id"]208+is_success = res.get("success", True)209+210+if not is_success:211+evaluation_results.append(212+EvaluationResult(213+request_id=req_id,214+request_body=res["request_body"],215+new_response=res.get("new_response"),216+valid_schema=False,217+quality_score=0.0,218+quality_feedback=f"Request failed: {res.get('error_type')} - {res.get('error_message')}",219+response_time=res.get("response_time"),220+success=False,221+error_type=res.get("error_type"),222+error_message=res.get("error_message"),223+ground_truth_match=False,224+ )225+ )226+continue227+228+pred = (res.get("new_response") or {}).get("ai_activity_type")229+230+if compare_to == "llm":231+if "ground_truth_llm" not in res:232+evaluation_results.append(233+EvaluationResult(234+request_id=req_id,235+request_body=res["request_body"],236+new_response=res.get("new_response"),237+valid_schema=res.get("valid_schema", True),238+quality_score=0.0,239+quality_feedback=(240+"Missing ground_truth_llm in run results. Ensure collect step populated it and "241+"endpoints.py passes it through."242+ ),243+response_time=res.get("response_time"),244+success=True,245+ground_truth_match=False,246+ )247+ )248+continue249+gt_label = res.get("ground_truth_llm")250+else:251+gt_path = self._ground_truths_dir / f"gt_{req_id}.json"252+if not gt_path.exists():253+evaluation_results.append(254+EvaluationResult(255+request_id=req_id,256+request_body=res["request_body"],257+new_response=res.get("new_response"),258+valid_schema=res.get("valid_schema", True),259+quality_score=0.0,260+quality_feedback=f"Ground truth file not found: {gt_path}",261+response_time=res.get("response_time"),262+success=True,263+ground_truth_match=False,264+ )265+ )266+continue267+268+with open(gt_path, "r") as f:269+ground_truth_dict: Dict[str, Any] = json.load(f)270+271+gt_label = (ground_truth_dict or {}).get("ai_activity_type")272+273+matched = pred == gt_label274+275+evaluation_results.append(276+EvaluationResult(277+request_id=req_id,278+request_body=res["request_body"],279+new_response=res.get("new_response"),280+valid_schema=res.get("valid_schema", True),281+quality_score=1.0 if matched else 0.0,282+quality_feedback="PASS" if matched else f"FAIL\nExpected: {gt_label}\nGot: {pred}",283+response_time=res.get("response_time"),284+success=True,285+ground_truth_match=matched,286+ )287+ )288+289+return evaluation_results290+291+292+if __name__ == "__main__":293+parser = create_parser(AiActivityTypeEvaluator)294+parser.add_argument(295+"--data-set",296+type=str,297+default="initial",298+dest="data_set",299+help="Dataset folder under eval_output/ai_activity_type/eval_datasets/ (default: initial)",300+ )301+302+parser.add_argument(303+"--populate-ground-truth-llm",304+action="store_true",305+dest="populate_ground_truth_llm",306+ )307+parser.add_argument(308+"--ground-truth-llm-endpoint",309+type=str,310+default="OPENAI:gpt-5_2",311+dest="ground_truth_llm_endpoint",312+ )313+314+parser.add_argument(315+"--compare-to-ground-truth",316+type=str,317+choices=["es", "llm"],318+default="es",319+dest="compare_to_ground_truth",320+ )321+322+args = parser.parse_args()323+AiActivityTypeEvaluator.DATA_SET = args.data_set324+325+AiActivityTypeEvaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_llm)326+AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpoint327+328+AiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truth329+330+asyncio.run(run_evaluation(AiActivityTypeEvaluator, args))</selection>” selected.
Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
Expand
Listen
Show more options
Gemini said
Gemini said
This Python script defines the
AiActivityTypeEvaluator
class, a specialized tool for evaluating AI-driven activity type classification. It extends a base evaluation framework to handle data collection, ground-truth comparison, and performance scoring.
Core Functionality
Core Functionality
Data Collection (
collect_test_set
):
* Supports fetching raw samples from
Elasticsearch (ES)
or loading them from
local JSON files
.
Implements a
replayability check
: It probes the live API endpoint to ensure the activity exists before including it in the dataset.
Filters data based on request body size, language, and the presence of an original response.
Ground Truth Management:
Saves inputs and "gold standard" responses to a local directory structure (
eval_output/ai_activity_type/...
).
Features an optional mode to populate ground truth using a high-order LLM (e.g.,
gpt-5_2
) if the
POPULATE_GROUND_TRUTH_LLM
flag is set.
Evaluation Logic (
evaluate_against_ground_truth
):
Compares the model's predicted
ai_activity_type
against either the original ES response or the LLM-generated ground truth.
Assigns a
quality_score
of
1.0
for a match and
0.0
for a mismatch or request failure.
Key Configuration Constants
Key Configuration Constants
Constant
Purpose
ENDPOINT_PATH
The specific API route being tested:
call/ai-activity-type
.
COMPARE_TO_GROUND_TRUTH
Determines if the baseline is the original data (
es
) or an
llm
.
SKIP_UNREPLAYABLE_ON_COLLECT
If
True
, discards records where the API returns "Cannot find activity."
REPLAYABILITY_CHECK_TIMEOUT_SECONDS
Limits the probe request to 20 seconds.
Constant
ENDPOINT_PATH
COMPARE_TO_GROUND_TRUTH
SKIP_UNREPLAYABLE_ON_COLLECT
REPLAYABILITY_CHECK_TIMEOUT_SECONDS
Purpose
The specific API route being tested:
call/ai-activity-type
.
Determines if the baseline is the original data (
es
) or an
llm
.
If
True
, discards records where the API returns "Cannot find activity."
Limits the probe request to 20 seconds.
Copy table
Execution Workflow
Execution Workflow
Parsing:
The script uses
create_parser
to handle CLI arguments like
--data-set
,
--populate-ground-truth-llm
, and
--compare-to-ground-truth
.
Initialization:
Sets up directory paths for inputs and ground truths based on the selected dataset name.
Run:
Executes the evaluation via
asyncio.run(run_evaluation(...))
, which orchestrates the collection, endpoint calling, and result scoring phases.
Copy prompt
Edit
You said are there soem issues with this PR
You said
are there soem issues with this PR
Show more options
Show thinking
Gemini said
Gemini said
Yes,
there are several issues and code smells in this Pull Request,
ranging from performance bottlenecks to architectural anti-patterns.
Here is a breakdown of the most notable issues,
categorized by impact:
Performance & Async Anti-Patterns
Performance & Async Anti-Patterns
Sequential Requests Defeating Async (Lines 86-106):
Inside
_collect_from_es
,
the code loops through
raw_samples
and
await
s a
session.post
request for each sample one by one.
This means the script waits for one HTTP request to finish before starting the next,
completely defeating the purpose of using
aiohttp
for asynchronous concurrency.
Fix:
Collect these coroutines into a list and use...
|
Firefox
|
Jy 19798 evaluation for ai activity types by nikol Jy 19798 evaluation for ai activity types by nikolaybiaivanov · Pull Request #468 · jiminny/prophet — Work...
|
github.com/jiminny/prophet/pull/468/changes#diff-d github.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8...
|
8543
|
|
JY-20543 add AJ reports User pilot tracking by Lak JY-20543 add AJ reports User pilot tracking by LakyLak · Pull Request #11932 · jiminny/app
Platform Sprint 1 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 1 Q2 - Platform Team - Scrum Board - Jira
Jy 19798 evaluation for ai activity types by nikolaybiaivanov · Pull Request #468 · jiminny/prophet
Jy 19798 evaluation for ai activity types by nikolaybiaivanov · Pull Request #468 · jiminny/prophet
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Copy prompt
You said I’m on page “<tabTitle>Jy 19798 evaluation for ai activity types by nikol</tabTitle>” with “<selection>@@ -0,0 +1,330 @@1+#!/usr/bin/env python32+import asyncio3+import json4+import os5+from pathlib import Path6+from typing import Any, Dict, List7+8+import aiohttp9+10+from src.component.call_processing.schemes.ai_activity_type_request import AiActivityTypeRequest11+from src.component.call_processing.schemes.ai_activity_type_response import AiActivityTypeResponse12+from src.eval.abstract_evaluator import AbstractEvaluator13+from src.eval.collectors import collect_test_set as es_collect_test_set14+from src.eval.common_evaluator_runner import create_parser, run_evaluation15+from src.eval.endpoints import run_endpoint as run_endpoint_http16+from src.eval.models import EvaluationResult17+18+19+class AiActivityTypeEvaluator(AbstractEvaluator):20+ENDPOINT_PATH = "call/ai-activity-type"21+REQUEST_MODEL = AiActivityTypeRequest22+RESPONSE_MODEL = AiActivityTypeResponse23+OUTPUT_SUBFOLDER = "ai_activity_type"24+USES_LLM_EVALUATION = False25+26+READS_FROM_DATASET_FOLDER = True27+28+DATA_SET: str = "initial"29+30+MINIMUM_REQUEST_BODY_SIZE: int = 031+32+SKIP_UNREPLAYABLE_ON_COLLECT: bool = True33+REPLAYABILITY_CHECK_TIMEOUT_SECONDS: float = 20.034+35+POPULATE_GROUND_TRUTH_LLM: bool = False36+GROUND_TRUTH_LLM_ENDPOINT: str = "OPENAI:gpt-5_2"37+38+COMPARE_TO_GROUND_TRUTH: str = "es"39+40+def __init__(self, **kwargs: Any) -> None:41+super().__init__(**kwargs)42+_repo_root = Path(__file__).parents[2]43+_datasets_root = _repo_root / "eval_output" / self.OUTPUT_SUBFOLDER / "eval_datasets"44+_dataset_dir = _datasets_root / self.__class__.DATA_SET45+self._dataset_dir = _dataset_dir46+self._inputs_dir = _dataset_dir / "input"47+self._ground_truths_dir = _dataset_dir / "ground_truth"48+49+async def collect_test_set(self) -> List[Dict[str, Any]]:50+if self.es_host_name:51+return await self._collect_from_es()52+return await self._collect_from_files()53+54+async def _collect_from_es(self) -> List[Dict[str, Any]]:55+if self._inputs_dir.exists() or self._ground_truths_dir.exists():56+raise FileExistsError(57+f"Dataset folder already exists: {self._inputs_dir.parent}. "58+"Choose a different --data-set name or remove the folder manually."59+ )60+61+raw_samples = await es_collect_test_set(62+es_host_name=self.es_host_name,63+es_port=self.es_port,64+endpoint_path=self.ENDPOINT_PATH,65+limit=self.limit,66+language_filter=self.language_filter,67+min_prompt_length=self.min_prompt_length,68+is_json_response=self.IS_JSON_RESPONSE,69+exclude_languages=self.exclude_languages,70+ )71+72+self._inputs_dir.mkdir(parents=True, exist_ok=True)73+self._ground_truths_dir.mkdir(parents=True, exist_ok=True)74+75+min_size = self.MINIMUM_REQUEST_BODY_SIZE76+samples: List[Dict[str, Any]] = []77+skipped_no_response = 078+skipped_too_small = 079+skipped_unreplayable = 080+skipped_probe_errors = 081+82+endpoint_url = f"{self.api_host}/{self.ENDPOINT_PATH}"83+84+timeout = aiohttp.ClientTimeout(total=self.REPLAYABILITY_CHECK_TIMEOUT_SECONDS)85+async with aiohttp.ClientSession(timeout=timeout) as session:86+for sample in raw_samples:87+es_id = sample["id"]88+request_body = sample["request_body"]89+original_response = sample.get("original_response")90+91+if not original_response:92+skipped_no_response += 193+continue94+95+if min_size and len(json.dumps(request_body)) < min_size:96+skipped_too_small += 197+continue98+99+if self.SKIP_UNREPLAYABLE_ON_COLLECT:100+try:101+async with session.post(endpoint_url, json=request_body) as resp:102+if resp.status != 200:103+body_text = await resp.text()104+if "Cannot find activity" in body_text:105+skipped_unreplayable += 1106+continue107+except Exception:108+skipped_probe_errors += 1109+110+with open(self._inputs_dir / f"{es_id}.json", "w") as f:111+json.dump(request_body, f, indent=2)112+113+with open(self._ground_truths_dir / f"gt_{es_id}.json", "w") as f:114+json.dump(original_response, f, indent=2)115+116+samples.append(117+ {118+"id": es_id,119+"request_body": request_body,120+"original_response": original_response,121+"ground_truth": original_response,122+ }123+ )124+125+print(f"Wrote {len(samples)} samples to {self._inputs_dir.parent}")126+if skipped_no_response:127+print(f"Skipped {skipped_no_response} records with no response body")128+if skipped_too_small:129+print(130+f"Skipped {skipped_too_small} records whose request body was smaller "131+f"than {min_size} characters"132+ )133+if skipped_unreplayable:134+print(f"Skipped {skipped_unreplayable} records that are not replayable (missing activity in ES)")135+if skipped_probe_errors:136+print(f"Warning: replayability probe errored for {skipped_probe_errors} records (kept those records)")137+138+if self.__class__.POPULATE_GROUND_TRUTH_LLM:139+label_results = await run_endpoint_http(140+test_set=samples,141+endpoint_url=endpoint_url,142+request_model=self.REQUEST_MODEL,143+force_llm_endpoint=self.__class__.GROUND_TRUTH_LLM_ENDPOINT,144+is_streaming_response=self.IS_STREAMING_RESPONSE,145+timeout_seconds=300.0,146+parallel_requests=self.parallel_requests,147+return_prompt_in_run=self.RETURN_PROMPT_IN_RUN,148+ )149+150+for i, r in enumerate(label_results):151+forced = self.__class__.GROUND_TRUTH_LLM_ENDPOINT152+model = forced.split(":", 1)[1] if ":" in forced else forced153+samples[i]["ground_truth_llm_model"] = model154+155+if not r.get("success", True):156+samples[i]["ground_truth_llm"] = None157+samples[i]["ground_truth_llm_reasoning"] = None158+samples[i]["ground_truth_llm_error_type"] = r.get("error_type")159+samples[i]["ground_truth_llm_error_message"] = r.get("error_message")160+continue161+162+resp = r.get("new_response") or {}163+samples[i]["ground_truth_llm"] = resp.get("ai_activity_type")164+samples[i]["ground_truth_llm_reasoning"] = resp.get("ai_activity_type_reasoning")165+166+return samples167+168+async def _collect_from_files(self) -> List[Dict[str, Any]]:169+if not self._inputs_dir.exists():170+raise FileNotFoundError(171+f"Dataset folder not found: {self._inputs_dir}. "172+"Run --steps collect first or check your --data-set name."173+ )174+175+samples: List[Dict[str, Any]] = []176+for filename in sorted(os.listdir(self._inputs_dir)):177+input_path = self._inputs_dir / filename178+gt_path = self._ground_truths_dir / f"gt_{filename}"179+180+if not gt_path.exists():181+print(f"Warning: ground truth not found for {filename}, skipping")182+continue183+184+with open(input_path, "r") as f:185+request_body: Dict[str, Any] = json.load(f)186+187+with open(gt_path, "r") as f:188+ground_truth: Dict[str, Any] = json.load(f)189+190+samples.append(191+ {192+"id": Path(filename).stem,193+"request_body": request_body,194+"original_response": None,195+"ground_truth": ground_truth,196+ }197+ )198+199+print(f"Loaded {len(samples)} samples from {self._inputs_dir.parent}")200+return samples201+202+async def evaluate_against_ground_truth(self, results: List[Dict[str, Any]]) -> List[EvaluationResult]:203+evaluation_results: List[EvaluationResult] = []204+compare_to = self.__class__.COMPARE_TO_GROUND_TRUTH205+206+for res in results:207+req_id = res["request_id"]208+is_success = res.get("success", True)209+210+if not is_success:211+evaluation_results.append(212+EvaluationResult(213+request_id=req_id,214+request_body=res["request_body"],215+new_response=res.get("new_response"),216+valid_schema=False,217+quality_score=0.0,218+quality_feedback=f"Request failed: {res.get('error_type')} - {res.get('error_message')}",219+response_time=res.get("response_time"),220+success=False,221+error_type=res.get("error_type"),222+error_message=res.get("error_message"),223+ground_truth_match=False,224+ )225+ )226+continue227+228+pred = (res.get("new_response") or {}).get("ai_activity_type")229+230+if compare_to == "llm":231+if "ground_truth_llm" not in res:232+evaluation_results.append(233+EvaluationResult(234+request_id=req_id,235+request_body=res["request_body"],236+new_response=res.get("new_response"),237+valid_schema=res.get("valid_schema", True),238+quality_score=0.0,239+quality_feedback=(240+"Missing ground_truth_llm in run results. Ensure collect step populated it and "241+"endpoints.py passes it through."242+ ),243+response_time=res.get("response_time"),244+success=True,245+ground_truth_match=False,246+ )247+ )248+continue249+gt_label = res.get("ground_truth_llm")250+else:251+gt_path = self._ground_truths_dir / f"gt_{req_id}.json"252+if not gt_path.exists():253+evaluation_results.append(254+EvaluationResult(255+request_id=req_id,256+request_body=res["request_body"],257+new_response=res.get("new_response"),258+valid_schema=res.get("valid_schema", True),259+quality_score=0.0,260+quality_feedback=f"Ground truth file not found: {gt_path}",261+response_time=res.get("response_time"),262+success=True,263+ground_truth_match=False,264+ )265+ )266+continue267+268+with open(gt_path, "r") as f:269+ground_truth_dict: Dict[str, Any] = json.load(f)270+271+gt_label = (ground_truth_dict or {}).get("ai_activity_type")272+273+matched = pred == gt_label274+275+evaluation_results.append(276+EvaluationResult(277+request_id=req_id,278+request_body=res["request_body"],279+new_response=res.get("new_response"),280+valid_schema=res.get("valid_schema", True),281+quality_score=1.0 if matched else 0.0,282+quality_feedback="PASS" if matched else f"FAIL\nExpected: {gt_label}\nGot: {pred}",283+response_time=res.get("response_time"),284+success=True,285+ground_truth_match=matched,286+ )287+ )288+289+return evaluation_results290+291+292+if __name__ == "__main__":293+parser = create_parser(AiActivityTypeEvaluator)294+parser.add_argument(295+"--data-set",296+type=str,297+default="initial",298+dest="data_set",299+help="Dataset folder under eval_output/ai_activity_type/eval_datasets/ (default: initial)",300+ )301+302+parser.add_argument(303+"--populate-ground-truth-llm",304+action="store_true",305+dest="populate_ground_truth_llm",306+ )307+parser.add_argument(308+"--ground-truth-llm-endpoint",309+type=str,310+default="OPENAI:gpt-5_2",311+dest="ground_truth_llm_endpoint",312+ )313+314+parser.add_argument(315+"--compare-to-ground-truth",316+type=str,317+choices=["es", "llm"],318+default="es",319+dest="compare_to_ground_truth",320+ )321+322+args = parser.parse_args()323+AiActivityTypeEvaluator.DATA_SET = args.data_set324+325+AiActivityTypeEvaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_llm)326+AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpoint327+328+AiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truth329+330+asyncio.run(run_evaluation(AiActivityTypeEvaluator, args))</selection>” selected. Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
You said
I’m on page “<tabTitle>Jy 19798 evaluation for ai activity types by nikol</tabTitle>” with “<selection>@@ -0,0 +1,330 @@1+#!/usr/bin/env python32+import asyncio3+import json4+import os5+from pathlib import Path6+from typing import Any, Dict, List7+8+import aiohttp9+10+from src.component.call_processing.schemes.ai_activity_type_request import AiActivityTypeRequest11+from src.component.call_processing.schemes.ai_activity_type_response import AiActivityTypeResponse12+from src.eval.abstract_evaluator import AbstractEvaluator13+from src.eval.collectors import collect_test_set as es_collect_test_set14+from src.eval.common_evaluator_runner import create_parser, run_evaluation15+from src.eval.endpoints import run_endpoint as run_endpoint_http16+from src.eval.models import EvaluationResult17+18+19+class AiActivityTypeEvaluator(AbstractEvaluator):20+ENDPOINT_PATH = "call/ai-activity-type"21+REQUEST_MODEL = AiActivityTypeRequest22+RESPONSE_MODEL = AiActivityTypeResponse23+OUTPUT_SUBFOLDER = "ai_activity_type"24+USES_LLM_EVALUATION = False25+26+READS_FROM_DATASET_FOLDER = True27+28+DATA_SET: str = "initial"29+30+MINIMUM_REQUEST_BODY_SIZE: int = 031+32+SKIP_UNREPLAYABLE_ON_COLLECT: bool = True33+REPLAYABILITY_CHECK_TIMEOUT_SECONDS: float = 20.034+35+POPULATE_GROUND_TRUTH_LLM: bool = False36+GROUND_TRUTH_LLM_ENDPOINT: str = "OPENAI:gpt-5_2"37+38+COMPARE_TO_GROUND_TRUTH: str = "es"39+40+def __init__(self, **kwargs: Any) -> None:41+super().__init__(**kwargs)42+_repo_root = Path(__file__).parents[2]43+_datasets_root = _repo_root / "eval_output" / self.OUTPUT_SUBFOLDER / "eval_datasets"44+_dataset_dir = _datasets_root / self.__class__.DATA_SET45+self._dataset_dir = _dataset_dir46+self._inputs_dir = _dataset_dir / "input"47+self._ground_truths_dir = _dataset_dir / "ground_truth"48+49+async def collect_test_set(self) -> List[Dict[str, Any]]:50+if self.es_host_name:51+return await self._collect_from_es()52+return await self._collect_from_files()53+54+async def _collect_from_es(self) -> List[Dict[str, Any]]:55+if self._inputs_dir.exists() or self._ground_truths_dir.exists():56+raise FileExistsError(57+f"Dataset folder already exists: {self._inputs_dir.parent}. "58+"Choose a different --data-set name or remove the folder manually."59+ )60+61+raw_samples = await es_collect_test_set(62+es_host_name=self.es_host_name,63+es_port=self.es_port,64+endpoint_path=self.ENDPOINT_PATH,65+limit=self.limit,66+language_filter=self.language_filter,67+min_prompt_length=self.min_prompt_length,68+is_json_response=self.IS_JSON_RESPONSE,69+exclude_languages=self.exclude_languages,70+ )71+72+self._inputs_dir.mkdir(parents=True, exist_ok=True)73+self._ground_truths_dir.mkdir(parents=True, exist_ok=True)74+75+min_size = self.MINIMUM_REQUEST_BODY_SIZE76+samples: List[Dict[str, Any]] = []77+skipped_no_response = 078+skipped_too_small = 079+skipped_unreplayable = 080+skipped_probe_errors = 081+82+endpoint_url = f"{self.api_host}/{self.ENDPOINT_PATH}"83+84+timeout = aiohttp.ClientTimeout(total=self.REPLAYABILITY_CHECK_TIMEOUT_SECONDS)85+async with aiohttp.ClientSession(timeout=timeout) as session:86+for sample in raw_samples:87+es_id = sample["id"]88+request_body = sample["request_body"]89+original_response = sample.get("original_response")90+91+if not original_response:92+skipped_no_response += 193+continue94+95+if min_size and len(json.dumps(request_body)) < min_size:96+skipped_too_small += 197+continue98+99+if self.SKIP_UNREPLAYABLE_ON_COLLECT:100+try:101+async with session.post(endpoint_url, json=request_body) as resp:102+if resp.status != 200:103+body_text = await resp.text()104+if "Cannot find activity" in body_text:105+skipped_unreplayable += 1106+continue107+except Exception:108+skipped_probe_errors += 1109+110+with open(self._inputs_dir / f"{es_id}.json", "w") as f:111+json.dump(request_body, f, indent=2)112+113+with open(self._ground_truths_dir / f"gt_{es_id}.json", "w") as f:114+json.dump(original_response, f, indent=2)115+116+samples.append(117+ {118+"id": es_id,119+"request_body": request_body,120+"original_response": original_response,121+"ground_truth": original_response,122+ }123+ )124+125+print(f"Wrote {len(samples)} samples to {self._inputs_dir.parent}")126+if skipped_no_response:127+print(f"Skipped {skipped_no_response} records with no response body")128+if skipped_too_small:129+print(130+f"Skipped {skipped_too_small} records whose request body was smaller "131+f"than {min_size} characters"132+ )133+if skipped_unreplayable:134+print(f"Skipped {skipped_unreplayable} records that are not replayable (missing activity in ES)")135+if skipped_probe_errors:136+print(f"Warning: replayability probe errored for {skipped_probe_errors} records (kept those records)")137+138+if self.__class__.POPULATE_GROUND_TRUTH_LLM:139+label_results = await run_endpoint_http(140+test_set=samples,141+endpoint_url=endpoint_url,142+request_model=self.REQUEST_MODEL,143+force_llm_endpoint=self.__class__.GROUND_TRUTH_LLM_ENDPOINT,144+is_streaming_response=self.IS_STREAMING_RESPONSE,145+timeout_seconds=300.0,146+parallel_requests=self.parallel_requests,147+return_prompt_in_run=self.RETURN_PROMPT_IN_RUN,148+ )149+150+for i, r in enumerate(label_results):151+forced = self.__class__.GROUND_TRUTH_LLM_ENDPOINT152+model = forced.split(":", 1)[1] if ":" in forced else forced153+samples[i]["ground_truth_llm_model"] = model154+155+if not r.get("success", True):156+samples[i]["ground_truth_llm"] = None157+samples[i]["ground_truth_llm_reasoning"] = None158+samples[i]["ground_truth_llm_error_type"] = r.get("error_type")159+samples[i]["ground_truth_llm_error_message"] = r.get("error_message")160+continue161+162+resp = r.get("new_response") or {}163+samples[i]["ground_truth_llm"] = resp.get("ai_activity_type")164+samples[i]["ground_truth_llm_reasoning"] = resp.get("ai_activity_type_reasoning")165+166+return samples167+168+async def _collect_from_files(self) -> List[Dict[str, Any]]:169+if not self._inputs_dir.exists():170+raise FileNotFoundError(171+f"Dataset folder not found: {self._inputs_dir}. "172+"Run --steps collect first or check your --data-set name."173+ )174+175+samples: List[Dict[str, Any]] = []176+for filename in sorted(os.listdir(self._inputs_dir)):177+input_path = self._inputs_dir / filename178+gt_path = self._ground_truths_dir / f"gt_{filename}"179+180+if not gt_path.exists():181+print(f"Warning: ground truth not found for {filename}, skipping")182+continue183+184+with open(input_path, "r") as f:185+request_body: Dict[str, Any] = json.load(f)186+187+with open(gt_path, "r") as f:188+ground_truth: Dict[str, Any] = json.load(f)189+190+samples.append(191+ {192+"id": Path(filename).stem,193+"request_body": request_body,194+"original_response": None,195+"ground_truth": ground_truth,196+ }197+ )198+199+print(f"Loaded {len(samples)} samples from {self._inputs_dir.parent}")200+return samples201+202+async def evaluate_against_ground_truth(self, results: List[Dict[str, Any]]) -> List[EvaluationResult]:203+evaluation_results: List[EvaluationResult] = []204+compare_to = self.__class__.COMPARE_TO_GROUND_TRUTH205+206+for res in results:207+req_id = res["request_id"]208+is_success = res.get("success", True)209+210+if not is_success:211+evaluation_results.append(212+EvaluationResult(213+request_id=req_id,214+request_body=res["request_body"],215+new_response=res.get("new_response"),216+valid_schema=False,217+quality_score=0.0,218+quality_feedback=f"Request failed: {res.get('error_type')} - {res.get('error_message')}",219+response_time=res.get("response_time"),220+success=False,221+error_type=res.get("error_type"),222+error_message=res.get("error_message"),223+ground_truth_match=False,224+ )225+ )226+continue227+228+pred = (res.get("new_response") or {}).get("ai_activity_type")229+230+if compare_to == "llm":231+if "ground_truth_llm" not in res:232+evaluation_results.append(233+EvaluationResult(234+request_id=req_id,235+request_body=res["request_body"],236+new_response=res.get("new_response"),237+valid_schema=res.get("valid_schema", True),238+quality_score=0.0,239+quality_feedback=(240+"Missing ground_truth_llm in run results. Ensure collect step populated it and "241+"endpoints.py passes it through."242+ ),243+response_time=res.get("response_time"),244+success=True,245+ground_truth_match=False,246+ )247+ )248+continue249+gt_label = res.get("ground_truth_llm")250+else:251+gt_path = self._ground_truths_dir / f"gt_{req_id}.json"252+if not gt_path.exists():253+evaluation_results.append(254+EvaluationResult(255+request_id=req_id,256+request_body=res["request_body"],257+new_response=res.get("new_response"),258+valid_schema=res.get("valid_schema", True),259+quality_score=0.0,260+quality_feedback=f"Ground truth file not found: {gt_path}",261+response_time=res.get("response_time"),262+success=True,263+ground_truth_match=False,264+ )265+ )266+continue267+268+with open(gt_path, "r") as f:269+ground_truth_dict: Dict[str, Any] = json.load(f)270+271+gt_label = (ground_truth_dict or {}).get("ai_activity_type")272+273+matched = pred == gt_label274+275+evaluation_results.append(276+EvaluationResult(277+request_id=req_id,278+request_body=res["request_body"],279+new_response=res.get("new_response"),280+valid_schema=res.get("valid_schema", True),281+quality_score=1.0 if matched else 0.0,282+quality_feedback="PASS" if matched else f"FAIL\nExpected: {gt_label}\nGot: {pred}",283+response_time=res.get("response_time"),284+success=True,285+ground_truth_match=matched,286+ )287+ )288+289+return evaluation_results290+291+292+if __name__ == "__main__":293+parser = create_parser(AiActivityTypeEvaluator)294+parser.add_argument(295+"--data-set",296+type=str,297+default="initial",298+dest="data_set",299+help="Dataset folder under eval_output/ai_activity_type/eval_datasets/ (default: initial)",300+ )301+302+parser.add_argument(303+"--populate-ground-truth-llm",304+action="store_true",305+dest="populate_ground_truth_llm",306+ )307+parser.add_argument(308+"--ground-truth-llm-endpoint",309+type=str,310+default="OPENAI:gpt-5_2",311+dest="ground_truth_llm_endpoint",312+ )313+314+parser.add_argument(315+"--compare-to-ground-truth",316+type=str,317+choices=["es", "llm"],318+default="es",319+dest="compare_to_ground_truth",320+ )321+322+args = parser.parse_args()323+AiActivityTypeEvaluator.DATA_SET = args.data_set324+325+AiActivityTypeEvaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_llm)326+AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpoint327+328+AiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truth329+330+asyncio.run(run_evaluation(AiActivityTypeEvaluator, args))</selection>” selected.
Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
Expand
Listen
Show more options
Gemini said
Gemini said
This Python script defines the
AiActivityTypeEvaluator
class, a specialized tool for evaluating AI-driven activity type classification. It extends a base evaluation framework to handle data collection, ground-truth comparison, and performance scoring.
Core Functionality
Core Functionality
Data Collection (
collect_test_set
):
* Supports fetching raw samples from
Elasticsearch (ES)
or loading them from
local JSON files
.
Implements a
replayability check
: It probes the live API endpoint to ensure the activity exists before including it in the dataset.
Filters data based on request body size, language, and the presence of an original response.
Ground Truth Management:
Saves inputs and "gold standard" responses to a local directory structure (
eval_output/ai_activity_type/...
).
Features an optional mode to populate ground truth using a high-order LLM (e.g.,
gpt-5_2
) if the
POPULATE_GROUND_TRUTH_LLM
flag is set.
Evaluation Logic (
evaluate_against_ground_truth
):
Compares the model's predicted
ai_activity_type
against either the original ES response or the LLM-generated ground truth.
Assigns a
quality_score
of
1.0
for a match and
0.0
for a mismatch or request failure.
Key Configuration Constants
Key Configuration Constants
Constant
Purpose
ENDPOINT_PATH
The specific API route being tested:
call/ai-activity-type
.
COMPARE_TO_GROUND_TRUTH
Determines if the baseline is the original data (
es
) or an
llm
.
SKIP_UNREPLAYABLE_ON_COLLECT
If
True
, discards records where the API returns "Cannot find activity."
REPLAYABILITY_CHECK_TIMEOUT_SECONDS
Limits the probe request to 20 seconds.
Constant
ENDPOINT_PATH
COMPARE_TO_GROUND_TRUTH
SKIP_UNREPLAYABLE_ON_COLLECT
REPLAYABILITY_CHECK_TIMEOUT_SECONDS
Purpose
The specific API route being tested:
call/ai-activity-type
.
Determines if the baseline is the original data (
es
) or an
llm
.
If
True
, discards records where the API returns "Cannot find activity."
Limits the probe request to 20 seconds.
Copy table
Execution Workflow
Execution Workflow
Parsing:
The script uses
create_parser
to handle CLI arguments like
--data-set
,
--populate-ground-truth-llm
, and
--compare-to-ground-truth
.
Initialization:
Sets up directory paths for inputs and ground truths based on the selected dataset name.
Run:
Executes the evaluation via
asyncio.run(run_evaluation(...))
, which orchestrates the collection, endpoint calling, and result scoring phases.
Copy prompt
Edit
You said are there soem issues with this PR
You said
are there soem issues with this PR
Show more options
Show thinking
Gemini said
Gemini said
Yes,
there are several issues and code smells in this Pull Request,
ranging from performance bottlenecks to architectural anti-patterns.
Here is a breakdown of the most notable issues,
categorized by impact:
Performance & Async Anti-Patterns
Performance & Async Anti-Patterns
Sequential Requests Defeating Async (Lines 86-106):
Inside
_collect_from_es
,
the code loops through
raw_samples
and
await
s a
session.post
request for each sample one by one.
This means the script waits for one HTTP request to finish before starting the next,
completely defeating the purpose of using
aiohttp
for asynchronous concurrency.
Fix:
Collect these coroutines into a list and use
asyncio.gather()
(ideally wrapped in an
asyncio.Semaphore
to avoid overwhelming the endpoint) to run the replayability probes concurrently.
Blocking File I/O in Async Functions (Lines 110-114, 184-188, 268-269):
Standard synchronous file operations (
with open(...)
,
json.dump
,
json.load
) are used inside
async def
functions.
In Python,
this blocks the entire event loop while the disk is reading/writing.
Fix:
Use an asynchronous file I/O library like
aiofiles
,
or run the blocking I/O operations in a separate thread using
asyncio.to_thread()...
|
Firefox
|
Jy 19798 evaluation for ai activity types by nikol Jy 19798 evaluation for ai activity types by nikolaybiaivanov · Pull Request #468 · jiminny/prophet — Work...
|
github.com/jiminny/prophet/pull/468/changes#diff-d github.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8...
|
8544
|
|
FirefoxFileEoitViewHistoryBookmarksProfilesToolsWi FirefoxFileEoitViewHistoryBookmarksProfilesToolsWindow Helpgithub.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8~ Google Gemini...l1 OpenJy 19798 evaluation for ai activity types #468All commitsPlatform Sprint 1 Q2 - Platform Tea@ Jy 19798 evaluation for ai activ+ New labnikolaybiaivanov wants to merge 9 commits into master from JY-19798-evaluation-for-ai-activity-types [v src/eval/eval-ai-activity-type.py SQ Filter files...• E eval_output/ai_activity_type/eval…..> • ground_truth>I inoutE src/evalE endpoints.pyE eval-ai-activity-type.py#.gitignore293294299are there soem issueswith this PRShow thinking vYes, there are several issues and code smellsin this Pull Request, ranging from performancebottlenecks to architectural anti-patterns.Here is a breakdown of the most notableissues, categorized by impact:Performance & Async Anti-Patterns• Sequential Requests Defeating Asyncluines ch-06t nsiee_collect_ from_es, the code loopsthrough raw_samples and await sasession.post request for each sampleone by one. This means the script waitsfor one HTTP request to finish beforestarting the next, completely defeatingthe purpose of using aiohttp forasynchronous concurrency.• Fix: Collect these coroutines into alist and use asyncio.gather()(ideally wrapped in anasyncio. Semaphore to avoidoverwhelming the endpoint) to runthe replayability probes concurrently.• Blocking File I/O in Async Functions(Lines 110-114, 184-188, 268-269):Standard synchronous file operations(with open(...), json. dump,json.load ) are used inside async def• Enter a prompt for Gemini+ProYour Jiminny chats aren't used to improve our models. Gemini is Alane can make mistakes, Inciualne aoout peodleYour privacy & GeminiSummarize page329330 Đ‹-0ll • | Daily - Platform • nowA100% CD8 • Tue 14 Apr 9:51:25• 2 1 203 viewed• Awaltind aporoval+330 UUUUu• Viewedparser = create_parser(AiActivityTypeEvaluator)parser.aad arcument"_-data-set",lyoe-sutndetault="initial",dest="data_set",nelo= Vataser Toler uncer eval outvurral acuvity tviereval catasets, etault.initial)",parser.aad arcument"--populate-ground-truth-1lm",action="store_true",dest="populate_ground_truth_llm",parser.add araument"--ground-truth-llm-endpoint",type=str,default="OPENAI: gpt-5_2",dest="ground_truth_llm_endpoint",parser.add_argument("--compare-to-ground-truth",type=str,choices=["es", "1lm"],default="es",dest="compare_to_ground_truth",aros = parser.parse arasAiActivityTypeEvaluator.DATA_SET = args.data_setA1Activitylypeevaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_Lm)AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpointAiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truthasyncio.run(run_evaluation(AiActivityTypeEvaluator, args))> .gitignore & ++4 0000• ViewedYou made it to the end!...
|
NULL
|
NULL
|
NULL
|
8545
|
|
JY-20543 add AJ reports User pilot tracking by Lak JY-20543 add AJ reports User pilot tracking by LakyLak · Pull Request #11932 · jiminny/app
Platform Sprint 1 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 1 Q2 - Platform Team - Scrum Board - Jira
Jy 19798 evaluation for ai activity types by nikolaybiaivanov · Pull Request #468 · jiminny/prophet
Jy 19798 evaluation for ai activity types by nikolaybiaivanov · Pull Request #468 · jiminny/prophet
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New Chat
Open menu for conversation actions.
Conversation with Gemini
Conversation with Gemini
Copy prompt
You said I’m on page “<tabTitle>Jy 19798 evaluation for ai activity types by nikol</tabTitle>” with “<selection>@@ -0,0 +1,330 @@1+#!/usr/bin/env python32+import asyncio3+import json4+import os5+from pathlib import Path6+from typing import Any, Dict, List7+8+import aiohttp9+10+from src.component.call_processing.schemes.ai_activity_type_request import AiActivityTypeRequest11+from src.component.call_processing.schemes.ai_activity_type_response import AiActivityTypeResponse12+from src.eval.abstract_evaluator import AbstractEvaluator13+from src.eval.collectors import collect_test_set as es_collect_test_set14+from src.eval.common_evaluator_runner import create_parser, run_evaluation15+from src.eval.endpoints import run_endpoint as run_endpoint_http16+from src.eval.models import EvaluationResult17+18+19+class AiActivityTypeEvaluator(AbstractEvaluator):20+ENDPOINT_PATH = "call/ai-activity-type"21+REQUEST_MODEL = AiActivityTypeRequest22+RESPONSE_MODEL = AiActivityTypeResponse23+OUTPUT_SUBFOLDER = "ai_activity_type"24+USES_LLM_EVALUATION = False25+26+READS_FROM_DATASET_FOLDER = True27+28+DATA_SET: str = "initial"29+30+MINIMUM_REQUEST_BODY_SIZE: int = 031+32+SKIP_UNREPLAYABLE_ON_COLLECT: bool = True33+REPLAYABILITY_CHECK_TIMEOUT_SECONDS: float = 20.034+35+POPULATE_GROUND_TRUTH_LLM: bool = False36+GROUND_TRUTH_LLM_ENDPOINT: str = "OPENAI:gpt-5_2"37+38+COMPARE_TO_GROUND_TRUTH: str = "es"39+40+def __init__(self, **kwargs: Any) -> None:41+super().__init__(**kwargs)42+_repo_root = Path(__file__).parents[2]43+_datasets_root = _repo_root / "eval_output" / self.OUTPUT_SUBFOLDER / "eval_datasets"44+_dataset_dir = _datasets_root / self.__class__.DATA_SET45+self._dataset_dir = _dataset_dir46+self._inputs_dir = _dataset_dir / "input"47+self._ground_truths_dir = _dataset_dir / "ground_truth"48+49+async def collect_test_set(self) -> List[Dict[str, Any]]:50+if self.es_host_name:51+return await self._collect_from_es()52+return await self._collect_from_files()53+54+async def _collect_from_es(self) -> List[Dict[str, Any]]:55+if self._inputs_dir.exists() or self._ground_truths_dir.exists():56+raise FileExistsError(57+f"Dataset folder already exists: {self._inputs_dir.parent}. "58+"Choose a different --data-set name or remove the folder manually."59+ )60+61+raw_samples = await es_collect_test_set(62+es_host_name=self.es_host_name,63+es_port=self.es_port,64+endpoint_path=self.ENDPOINT_PATH,65+limit=self.limit,66+language_filter=self.language_filter,67+min_prompt_length=self.min_prompt_length,68+is_json_response=self.IS_JSON_RESPONSE,69+exclude_languages=self.exclude_languages,70+ )71+72+self._inputs_dir.mkdir(parents=True, exist_ok=True)73+self._ground_truths_dir.mkdir(parents=True, exist_ok=True)74+75+min_size = self.MINIMUM_REQUEST_BODY_SIZE76+samples: List[Dict[str, Any]] = []77+skipped_no_response = 078+skipped_too_small = 079+skipped_unreplayable = 080+skipped_probe_errors = 081+82+endpoint_url = f"{self.api_host}/{self.ENDPOINT_PATH}"83+84+timeout = aiohttp.ClientTimeout(total=self.REPLAYABILITY_CHECK_TIMEOUT_SECONDS)85+async with aiohttp.ClientSession(timeout=timeout) as session:86+for sample in raw_samples:87+es_id = sample["id"]88+request_body = sample["request_body"]89+original_response = sample.get("original_response")90+91+if not original_response:92+skipped_no_response += 193+continue94+95+if min_size and len(json.dumps(request_body)) < min_size:96+skipped_too_small += 197+continue98+99+if self.SKIP_UNREPLAYABLE_ON_COLLECT:100+try:101+async with session.post(endpoint_url, json=request_body) as resp:102+if resp.status != 200:103+body_text = await resp.text()104+if "Cannot find activity" in body_text:105+skipped_unreplayable += 1106+continue107+except Exception:108+skipped_probe_errors += 1109+110+with open(self._inputs_dir / f"{es_id}.json", "w") as f:111+json.dump(request_body, f, indent=2)112+113+with open(self._ground_truths_dir / f"gt_{es_id}.json", "w") as f:114+json.dump(original_response, f, indent=2)115+116+samples.append(117+ {118+"id": es_id,119+"request_body": request_body,120+"original_response": original_response,121+"ground_truth": original_response,122+ }123+ )124+125+print(f"Wrote {len(samples)} samples to {self._inputs_dir.parent}")126+if skipped_no_response:127+print(f"Skipped {skipped_no_response} records with no response body")128+if skipped_too_small:129+print(130+f"Skipped {skipped_too_small} records whose request body was smaller "131+f"than {min_size} characters"132+ )133+if skipped_unreplayable:134+print(f"Skipped {skipped_unreplayable} records that are not replayable (missing activity in ES)")135+if skipped_probe_errors:136+print(f"Warning: replayability probe errored for {skipped_probe_errors} records (kept those records)")137+138+if self.__class__.POPULATE_GROUND_TRUTH_LLM:139+label_results = await run_endpoint_http(140+test_set=samples,141+endpoint_url=endpoint_url,142+request_model=self.REQUEST_MODEL,143+force_llm_endpoint=self.__class__.GROUND_TRUTH_LLM_ENDPOINT,144+is_streaming_response=self.IS_STREAMING_RESPONSE,145+timeout_seconds=300.0,146+parallel_requests=self.parallel_requests,147+return_prompt_in_run=self.RETURN_PROMPT_IN_RUN,148+ )149+150+for i, r in enumerate(label_results):151+forced = self.__class__.GROUND_TRUTH_LLM_ENDPOINT152+model = forced.split(":", 1)[1] if ":" in forced else forced153+samples[i]["ground_truth_llm_model"] = model154+155+if not r.get("success", True):156+samples[i]["ground_truth_llm"] = None157+samples[i]["ground_truth_llm_reasoning"] = None158+samples[i]["ground_truth_llm_error_type"] = r.get("error_type")159+samples[i]["ground_truth_llm_error_message"] = r.get("error_message")160+continue161+162+resp = r.get("new_response") or {}163+samples[i]["ground_truth_llm"] = resp.get("ai_activity_type")164+samples[i]["ground_truth_llm_reasoning"] = resp.get("ai_activity_type_reasoning")165+166+return samples167+168+async def _collect_from_files(self) -> List[Dict[str, Any]]:169+if not self._inputs_dir.exists():170+raise FileNotFoundError(171+f"Dataset folder not found: {self._inputs_dir}. "172+"Run --steps collect first or check your --data-set name."173+ )174+175+samples: List[Dict[str, Any]] = []176+for filename in sorted(os.listdir(self._inputs_dir)):177+input_path = self._inputs_dir / filename178+gt_path = self._ground_truths_dir / f"gt_{filename}"179+180+if not gt_path.exists():181+print(f"Warning: ground truth not found for {filename}, skipping")182+continue183+184+with open(input_path, "r") as f:185+request_body: Dict[str, Any] = json.load(f)186+187+with open(gt_path, "r") as f:188+ground_truth: Dict[str, Any] = json.load(f)189+190+samples.append(191+ {192+"id": Path(filename).stem,193+"request_body": request_body,194+"original_response": None,195+"ground_truth": ground_truth,196+ }197+ )198+199+print(f"Loaded {len(samples)} samples from {self._inputs_dir.parent}")200+return samples201+202+async def evaluate_against_ground_truth(self, results: List[Dict[str, Any]]) -> List[EvaluationResult]:203+evaluation_results: List[EvaluationResult] = []204+compare_to = self.__class__.COMPARE_TO_GROUND_TRUTH205+206+for res in results:207+req_id = res["request_id"]208+is_success = res.get("success", True)209+210+if not is_success:211+evaluation_results.append(212+EvaluationResult(213+request_id=req_id,214+request_body=res["request_body"],215+new_response=res.get("new_response"),216+valid_schema=False,217+quality_score=0.0,218+quality_feedback=f"Request failed: {res.get('error_type')} - {res.get('error_message')}",219+response_time=res.get("response_time"),220+success=False,221+error_type=res.get("error_type"),222+error_message=res.get("error_message"),223+ground_truth_match=False,224+ )225+ )226+continue227+228+pred = (res.get("new_response") or {}).get("ai_activity_type")229+230+if compare_to == "llm":231+if "ground_truth_llm" not in res:232+evaluation_results.append(233+EvaluationResult(234+request_id=req_id,235+request_body=res["request_body"],236+new_response=res.get("new_response"),237+valid_schema=res.get("valid_schema", True),238+quality_score=0.0,239+quality_feedback=(240+"Missing ground_truth_llm in run results. Ensure collect step populated it and "241+"endpoints.py passes it through."242+ ),243+response_time=res.get("response_time"),244+success=True,245+ground_truth_match=False,246+ )247+ )248+continue249+gt_label = res.get("ground_truth_llm")250+else:251+gt_path = self._ground_truths_dir / f"gt_{req_id}.json"252+if not gt_path.exists():253+evaluation_results.append(254+EvaluationResult(255+request_id=req_id,256+request_body=res["request_body"],257+new_response=res.get("new_response"),258+valid_schema=res.get("valid_schema", True),259+quality_score=0.0,260+quality_feedback=f"Ground truth file not found: {gt_path}",261+response_time=res.get("response_time"),262+success=True,263+ground_truth_match=False,264+ )265+ )266+continue267+268+with open(gt_path, "r") as f:269+ground_truth_dict: Dict[str, Any] = json.load(f)270+271+gt_label = (ground_truth_dict or {}).get("ai_activity_type")272+273+matched = pred == gt_label274+275+evaluation_results.append(276+EvaluationResult(277+request_id=req_id,278+request_body=res["request_body"],279+new_response=res.get("new_response"),280+valid_schema=res.get("valid_schema", True),281+quality_score=1.0 if matched else 0.0,282+quality_feedback="PASS" if matched else f"FAIL\nExpected: {gt_label}\nGot: {pred}",283+response_time=res.get("response_time"),284+success=True,285+ground_truth_match=matched,286+ )287+ )288+289+return evaluation_results290+291+292+if __name__ == "__main__":293+parser = create_parser(AiActivityTypeEvaluator)294+parser.add_argument(295+"--data-set",296+type=str,297+default="initial",298+dest="data_set",299+help="Dataset folder under eval_output/ai_activity_type/eval_datasets/ (default: initial)",300+ )301+302+parser.add_argument(303+"--populate-ground-truth-llm",304+action="store_true",305+dest="populate_ground_truth_llm",306+ )307+parser.add_argument(308+"--ground-truth-llm-endpoint",309+type=str,310+default="OPENAI:gpt-5_2",311+dest="ground_truth_llm_endpoint",312+ )313+314+parser.add_argument(315+"--compare-to-ground-truth",316+type=str,317+choices=["es", "llm"],318+default="es",319+dest="compare_to_ground_truth",320+ )321+322+args = parser.parse_args()323+AiActivityTypeEvaluator.DATA_SET = args.data_set324+325+AiActivityTypeEvaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_llm)326+AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpoint327+328+AiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truth329+330+asyncio.run(run_evaluation(AiActivityTypeEvaluator, args))</selection>” selected. Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
You said
I’m on page “<tabTitle>Jy 19798 evaluation for ai activity types by nikol</tabTitle>” with “<selection>@@ -0,0 +1,330 @@1+#!/usr/bin/env python32+import asyncio3+import json4+import os5+from pathlib import Path6+from typing import Any, Dict, List7+8+import aiohttp9+10+from src.component.call_processing.schemes.ai_activity_type_request import AiActivityTypeRequest11+from src.component.call_processing.schemes.ai_activity_type_response import AiActivityTypeResponse12+from src.eval.abstract_evaluator import AbstractEvaluator13+from src.eval.collectors import collect_test_set as es_collect_test_set14+from src.eval.common_evaluator_runner import create_parser, run_evaluation15+from src.eval.endpoints import run_endpoint as run_endpoint_http16+from src.eval.models import EvaluationResult17+18+19+class AiActivityTypeEvaluator(AbstractEvaluator):20+ENDPOINT_PATH = "call/ai-activity-type"21+REQUEST_MODEL = AiActivityTypeRequest22+RESPONSE_MODEL = AiActivityTypeResponse23+OUTPUT_SUBFOLDER = "ai_activity_type"24+USES_LLM_EVALUATION = False25+26+READS_FROM_DATASET_FOLDER = True27+28+DATA_SET: str = "initial"29+30+MINIMUM_REQUEST_BODY_SIZE: int = 031+32+SKIP_UNREPLAYABLE_ON_COLLECT: bool = True33+REPLAYABILITY_CHECK_TIMEOUT_SECONDS: float = 20.034+35+POPULATE_GROUND_TRUTH_LLM: bool = False36+GROUND_TRUTH_LLM_ENDPOINT: str = "OPENAI:gpt-5_2"37+38+COMPARE_TO_GROUND_TRUTH: str = "es"39+40+def __init__(self, **kwargs: Any) -> None:41+super().__init__(**kwargs)42+_repo_root = Path(__file__).parents[2]43+_datasets_root = _repo_root / "eval_output" / self.OUTPUT_SUBFOLDER / "eval_datasets"44+_dataset_dir = _datasets_root / self.__class__.DATA_SET45+self._dataset_dir = _dataset_dir46+self._inputs_dir = _dataset_dir / "input"47+self._ground_truths_dir = _dataset_dir / "ground_truth"48+49+async def collect_test_set(self) -> List[Dict[str, Any]]:50+if self.es_host_name:51+return await self._collect_from_es()52+return await self._collect_from_files()53+54+async def _collect_from_es(self) -> List[Dict[str, Any]]:55+if self._inputs_dir.exists() or self._ground_truths_dir.exists():56+raise FileExistsError(57+f"Dataset folder already exists: {self._inputs_dir.parent}. "58+"Choose a different --data-set name or remove the folder manually."59+ )60+61+raw_samples = await es_collect_test_set(62+es_host_name=self.es_host_name,63+es_port=self.es_port,64+endpoint_path=self.ENDPOINT_PATH,65+limit=self.limit,66+language_filter=self.language_filter,67+min_prompt_length=self.min_prompt_length,68+is_json_response=self.IS_JSON_RESPONSE,69+exclude_languages=self.exclude_languages,70+ )71+72+self._inputs_dir.mkdir(parents=True, exist_ok=True)73+self._ground_truths_dir.mkdir(parents=True, exist_ok=True)74+75+min_size = self.MINIMUM_REQUEST_BODY_SIZE76+samples: List[Dict[str, Any]] = []77+skipped_no_response = 078+skipped_too_small = 079+skipped_unreplayable = 080+skipped_probe_errors = 081+82+endpoint_url = f"{self.api_host}/{self.ENDPOINT_PATH}"83+84+timeout = aiohttp.ClientTimeout(total=self.REPLAYABILITY_CHECK_TIMEOUT_SECONDS)85+async with aiohttp.ClientSession(timeout=timeout) as session:86+for sample in raw_samples:87+es_id = sample["id"]88+request_body = sample["request_body"]89+original_response = sample.get("original_response")90+91+if not original_response:92+skipped_no_response += 193+continue94+95+if min_size and len(json.dumps(request_body)) < min_size:96+skipped_too_small += 197+continue98+99+if self.SKIP_UNREPLAYABLE_ON_COLLECT:100+try:101+async with session.post(endpoint_url, json=request_body) as resp:102+if resp.status != 200:103+body_text = await resp.text()104+if "Cannot find activity" in body_text:105+skipped_unreplayable += 1106+continue107+except Exception:108+skipped_probe_errors += 1109+110+with open(self._inputs_dir / f"{es_id}.json", "w") as f:111+json.dump(request_body, f, indent=2)112+113+with open(self._ground_truths_dir / f"gt_{es_id}.json", "w") as f:114+json.dump(original_response, f, indent=2)115+116+samples.append(117+ {118+"id": es_id,119+"request_body": request_body,120+"original_response": original_response,121+"ground_truth": original_response,122+ }123+ )124+125+print(f"Wrote {len(samples)} samples to {self._inputs_dir.parent}")126+if skipped_no_response:127+print(f"Skipped {skipped_no_response} records with no response body")128+if skipped_too_small:129+print(130+f"Skipped {skipped_too_small} records whose request body was smaller "131+f"than {min_size} characters"132+ )133+if skipped_unreplayable:134+print(f"Skipped {skipped_unreplayable} records that are not replayable (missing activity in ES)")135+if skipped_probe_errors:136+print(f"Warning: replayability probe errored for {skipped_probe_errors} records (kept those records)")137+138+if self.__class__.POPULATE_GROUND_TRUTH_LLM:139+label_results = await run_endpoint_http(140+test_set=samples,141+endpoint_url=endpoint_url,142+request_model=self.REQUEST_MODEL,143+force_llm_endpoint=self.__class__.GROUND_TRUTH_LLM_ENDPOINT,144+is_streaming_response=self.IS_STREAMING_RESPONSE,145+timeout_seconds=300.0,146+parallel_requests=self.parallel_requests,147+return_prompt_in_run=self.RETURN_PROMPT_IN_RUN,148+ )149+150+for i, r in enumerate(label_results):151+forced = self.__class__.GROUND_TRUTH_LLM_ENDPOINT152+model = forced.split(":", 1)[1] if ":" in forced else forced153+samples[i]["ground_truth_llm_model"] = model154+155+if not r.get("success", True):156+samples[i]["ground_truth_llm"] = None157+samples[i]["ground_truth_llm_reasoning"] = None158+samples[i]["ground_truth_llm_error_type"] = r.get("error_type")159+samples[i]["ground_truth_llm_error_message"] = r.get("error_message")160+continue161+162+resp = r.get("new_response") or {}163+samples[i]["ground_truth_llm"] = resp.get("ai_activity_type")164+samples[i]["ground_truth_llm_reasoning"] = resp.get("ai_activity_type_reasoning")165+166+return samples167+168+async def _collect_from_files(self) -> List[Dict[str, Any]]:169+if not self._inputs_dir.exists():170+raise FileNotFoundError(171+f"Dataset folder not found: {self._inputs_dir}. "172+"Run --steps collect first or check your --data-set name."173+ )174+175+samples: List[Dict[str, Any]] = []176+for filename in sorted(os.listdir(self._inputs_dir)):177+input_path = self._inputs_dir / filename178+gt_path = self._ground_truths_dir / f"gt_{filename}"179+180+if not gt_path.exists():181+print(f"Warning: ground truth not found for {filename}, skipping")182+continue183+184+with open(input_path, "r") as f:185+request_body: Dict[str, Any] = json.load(f)186+187+with open(gt_path, "r") as f:188+ground_truth: Dict[str, Any] = json.load(f)189+190+samples.append(191+ {192+"id": Path(filename).stem,193+"request_body": request_body,194+"original_response": None,195+"ground_truth": ground_truth,196+ }197+ )198+199+print(f"Loaded {len(samples)} samples from {self._inputs_dir.parent}")200+return samples201+202+async def evaluate_against_ground_truth(self, results: List[Dict[str, Any]]) -> List[EvaluationResult]:203+evaluation_results: List[EvaluationResult] = []204+compare_to = self.__class__.COMPARE_TO_GROUND_TRUTH205+206+for res in results:207+req_id = res["request_id"]208+is_success = res.get("success", True)209+210+if not is_success:211+evaluation_results.append(212+EvaluationResult(213+request_id=req_id,214+request_body=res["request_body"],215+new_response=res.get("new_response"),216+valid_schema=False,217+quality_score=0.0,218+quality_feedback=f"Request failed: {res.get('error_type')} - {res.get('error_message')}",219+response_time=res.get("response_time"),220+success=False,221+error_type=res.get("error_type"),222+error_message=res.get("error_message"),223+ground_truth_match=False,224+ )225+ )226+continue227+228+pred = (res.get("new_response") or {}).get("ai_activity_type")229+230+if compare_to == "llm":231+if "ground_truth_llm" not in res:232+evaluation_results.append(233+EvaluationResult(234+request_id=req_id,235+request_body=res["request_body"],236+new_response=res.get("new_response"),237+valid_schema=res.get("valid_schema", True),238+quality_score=0.0,239+quality_feedback=(240+"Missing ground_truth_llm in run results. Ensure collect step populated it and "241+"endpoints.py passes it through."242+ ),243+response_time=res.get("response_time"),244+success=True,245+ground_truth_match=False,246+ )247+ )248+continue249+gt_label = res.get("ground_truth_llm")250+else:251+gt_path = self._ground_truths_dir / f"gt_{req_id}.json"252+if not gt_path.exists():253+evaluation_results.append(254+EvaluationResult(255+request_id=req_id,256+request_body=res["request_body"],257+new_response=res.get("new_response"),258+valid_schema=res.get("valid_schema", True),259+quality_score=0.0,260+quality_feedback=f"Ground truth file not found: {gt_path}",261+response_time=res.get("response_time"),262+success=True,263+ground_truth_match=False,264+ )265+ )266+continue267+268+with open(gt_path, "r") as f:269+ground_truth_dict: Dict[str, Any] = json.load(f)270+271+gt_label = (ground_truth_dict or {}).get("ai_activity_type")272+273+matched = pred == gt_label274+275+evaluation_results.append(276+EvaluationResult(277+request_id=req_id,278+request_body=res["request_body"],279+new_response=res.get("new_response"),280+valid_schema=res.get("valid_schema", True),281+quality_score=1.0 if matched else 0.0,282+quality_feedback="PASS" if matched else f"FAIL\nExpected: {gt_label}\nGot: {pred}",283+response_time=res.get("response_time"),284+success=True,285+ground_truth_match=matched,286+ )287+ )288+289+return evaluation_results290+291+292+if __name__ == "__main__":293+parser = create_parser(AiActivityTypeEvaluator)294+parser.add_argument(295+"--data-set",296+type=str,297+default="initial",298+dest="data_set",299+help="Dataset folder under eval_output/ai_activity_type/eval_datasets/ (default: initial)",300+ )301+302+parser.add_argument(303+"--populate-ground-truth-llm",304+action="store_true",305+dest="populate_ground_truth_llm",306+ )307+parser.add_argument(308+"--ground-truth-llm-endpoint",309+type=str,310+default="OPENAI:gpt-5_2",311+dest="ground_truth_llm_endpoint",312+ )313+314+parser.add_argument(315+"--compare-to-ground-truth",316+type=str,317+choices=["es", "llm"],318+default="es",319+dest="compare_to_ground_truth",320+ )321+322+args = parser.parse_args()323+AiActivityTypeEvaluator.DATA_SET = args.data_set324+325+AiActivityTypeEvaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_llm)326+AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpoint327+328+AiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truth329+330+asyncio.run(run_evaluation(AiActivityTypeEvaluator, args))</selection>” selected.
Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy.
Expand
Listen
Show more options
Gemini said
Gemini said
This Python script defines the
AiActivityTypeEvaluator
class, a specialized tool for evaluating AI-driven activity type classification. It extends a base evaluation framework to handle data collection, ground-truth comparison, and performance scoring.
Core Functionality
Core Functionality
Data Collection (
collect_test_set
):
* Supports fetching raw samples from
Elasticsearch (ES)
or loading them from
local JSON files
.
Implements a
replayability check
: It probes the live API endpoint to ensure the activity exists before including it in the dataset.
Filters data based on request body size, language, and the presence of an original response.
Ground Truth Management:
Saves inputs and "gold standard" responses to a local directory structure (
eval_output/ai_activity_type/...
).
Features an optional mode to populate ground truth using a high-order LLM (e.g.,
gpt-5_2
) if the
POPULATE_GROUND_TRUTH_LLM
flag is set.
Evaluation Logic (
evaluate_against_ground_truth
):
Compares the model's predicted
ai_activity_type
against either the original ES response or the LLM-generated ground truth.
Assigns a
quality_score
of
1.0
for a match and
0.0
for a mismatch or request failure.
Key Configuration Constants
Key Configuration Constants
Constant
Purpose
ENDPOINT_PATH
The specific API route being tested:
call/ai-activity-type
.
COMPARE_TO_GROUND_TRUTH
Determines if the baseline is the original data (
es
) or an
llm
.
SKIP_UNREPLAYABLE_ON_COLLECT
If
True
, discards records where the API returns "Cannot find activity."
REPLAYABILITY_CHECK_TIMEOUT_SECONDS
Limits the probe request to 20 seconds.
Constant
ENDPOINT_PATH
COMPARE_TO_GROUND_TRUTH
SKIP_UNREPLAYABLE_ON_COLLECT
REPLAYABILITY_CHECK_TIMEOUT_SECONDS
Purpose
The specific API route being tested:
call/ai-activity-type
.
Determines if the baseline is the original data (
es
) or an
llm
.
If
True
, discards records where the API returns "Cannot find activity."
Limits the probe request to 20 seconds.
Copy table
Execution Workflow
Execution Workflow
Parsing:
The script uses
create_parser
to handle CLI arguments like
--data-set
,
--populate-ground-truth-llm
, and
--compare-to-ground-truth
.
Initialization:
Sets up directory paths for inputs and ground truths based on the selected dataset name.
Run:
Executes the evaluation via
asyncio.run(run_evaluation(...))
, which orchestrates the collection, endpoint calling, and result scoring phases.
Copy prompt
Edit
You said are there soem issues with this PR
You said
are there soem issues with this PR
Listen
Show more options
Show thinking
Gemini said
Gemini said
Yes, there are several issues and code smells in this Pull Request, ranging from performance bottlenecks to architectural anti-patterns.
Here is a breakdown of the most notable issues, categorized by impact:
Performance & Async Anti-Patterns
Performance & Async Anti-Patterns
Sequential Requests Defeating Async (Lines 86-106):
Inside
_collect_from_es
, the code loops through
raw_samples
and
await
s a
session.post
request for each sample one by one. This means the script waits for one HTTP request to finish before starting the next, completely defeating the purpose of using
aiohttp
for asynchronous concurrency.
Fix:
Collect these coroutines into a list and use
asyncio.gather()
(ideally wrapped in an
asyncio.Semaphore
to avoid overwhelming the endpoint) to run the replayability probes concurrently.
Blocking File I/O in Async Functions (Lines 110-114, 184-188, 268-269):
Standard synchronous file operations (
with open(...)
,
json.dump
,
json.load
) are used inside
async def
functions. In Python, this blocks the entire event loop while the disk is reading/writing.
Fix:
Use an asynchronous file I/O library like
aiofiles
, or run the blocking I/O operations in a separate thread using
asyncio.to_thread()
.
Architecture & State Management
Architecture & State Management
Mutating Class-Level Variables (Lines 323-328):
The script parses CLI arguments and directly mutates the class attributes (e.g.,
AiActivityTypeEvaluator.DATA_SET = args.data_set
). This is a major anti-pattern. If you ever try to run tests in parallel or create multiple instances of this evaluator in a single process, the state will bleed across them.
Fix:
Pass these arguments into the
__init__
constructor of the instance (
self.data_set = args.data_set
) rather than modifying the class definition globally.
Code Quality & Edge Cases
Code Quality & Edge Cases
Broad Exception Catching (Line 107):
Using a bare
except Exception:
during the replayability probe is dangerous. It will catch
aiohttp.ClientError
(which is intended), but it will also silently swallow
TimeoutError
or even memory errors, making debugging incredibly difficult if the endpoint goes down.
Fix:
Scope the exception to specific expected errors, like
except (aiohttp.ClientError, asyncio.TimeoutError):
.
Inefficient Size Calculation (Line 95):
len(json.dumps(request_body)) < min_size
requires serializing the entire dictionary to a string
just
to check its length. Since
min_size
defaults to
0
, the
if min_size
short-circuit saves it for now, but if anyone sets a minimum size, this will become an unnecessary CPU bottleneck on large datasets.
Potential Path Traversal Risk (Lines 110, 113, 251):
The script writes and reads files directly using...
|
Firefox
|
Jy 19798 evaluation for ai activity types by nikol Jy 19798 evaluation for ai activity types by nikolaybiaivanov · Pull Request #468 · jiminny/prophet — Work...
|
github.com/jiminny/prophet/pull/468/changes#diff-d github.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8...
|
8546
|
|
FirefoxFileEoitViewHistoryBookmarksProfilesToolsWi FirefoxFileEoitViewHistoryBookmarksProfilesToolsWindow Helpgithub.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8~ Google Gemini...l1 OpenJy 19798 evaluation for ai activity types #468All commitsPlatform Sprint 1 Q2 - Platform Tea@ Jy 19798 evaluation for ai activ+ New labnikolaybiaivanov wants to merge 9 commits into master from JY-19798-evaluation-for-ai-activity-types [v src/eval/eval-ai-activity-type.py SQ Filter files...• E eval_output/ai_activity_type/eval…..> • ground_truth>I inoutE src/eval* endpoints.pyE eval-ai-activity-type.pyE .gitignore293294299are there soem issueswith this PRShow thinking vYes, there are several issues and code smellsin this Pull Request, ranging from performancebottlenecks to architectural anti-patterns.Here is a breakdown of the most notableissues, categorized by impact:Performance & Async Anti-Patterns• Sequential Requests Defeating AsyncTuines ch-06nsioe_collect_ from_es, the code loopsthrough raw_samples and await sasession.post request for each sampleone by one. This means the script waitsfor one HTTP request to finish beforestarting the next, completely defeatingthe purpose of using aiohttp forasynchronous concurrency.• Fix: Collect these coroutines into alist and use asyncio.gather()(ideally wrapped in anasyncio. Semaphore to avoidoverwhelming the endpoint) to runthe replayability probes concurrently.• Blocking File I/O in Async Functions(Lines 110-114, 184-188, 268-269):Standard synchronous file operations(with open(...), json. dump,json.load ) are used inside async def• Enter a prompt for Gemini+Pro vYour Jiminny chats aren't used to improve our models. Gemini is Alane can make mistakes, Inciualne aoout peodleYour privacy & GeminiSummarize page329330 +‹-0ll • | Daily - Platform • nowA100% CD8 • Tue 14 Apr 9:51:26• 2 1 203 viewed• Awaltind aporoval+330 UUUUu• Viewedparser = create_parser(AiActivityTypeEvaluator)parser.aad arcument"_-data-set",lyoe-sutndetault="initial",dest="data_set",nelo= Vataser Toler uncer eval outvurral acuvity tviereval catasets, etault.initial)",parser.aad arcument"--populate-ground-truth-1lm",action="store_true",dest="populate_ground_truth_llm",parser.add araument"--ground-truth-llm-endpoint",type=str,default="OPENAI: gpt-5_2",dest="ground_truth_llm_endpoint",parser.add_argument("--compare-to-ground-truth",type=str,choices=["es", "1lm"],default="es",dest="compare_to_ground_truth",aros = parser.parse arasAiActivityTypeEvaluator.DATA_SET = args.data_setA1Activitylypeevaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_Lm)AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpointAiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truthasyncio.run(run_evaluation(AiActivityTypeEvaluator, args))> .gitignore & ++4 0000• ViewedYou made it to the end!...
|
NULL
|
NULL
|
NULL
|
8547
|
|
FirefoxFileEoitViewHistoryBookmarksProfilesToolsWi FirefoxFileEoitViewHistoryBookmarksProfilesToolsWindow Helpgithub.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8~ Google Gemini...l1 OpenJy 19798 evaluation for ai activity types #468All commitsPlatform Sprint 1 Q2 - Platform Tea@ Jy 19798 evaluation for ai activ+ New labnikolaybiaivanov wants to merge 9 commits into master from JY-19798-evaluation-for-ai-activity-types [v src/eval/eval-ai-activity-type.py SQ Filter files...• E eval_output/ai_activity_type/eval…..> • ground_truth>I inoutE src/evalE endpoints.pyE eval-ai-activity-type.py#.gitignore293294299are there soem issueswith this PRShow thinking vYes, there are several issues and code smellsin this Pull Request, ranging from performancebottlenecks to architectural anti-patterns.Here is a breakdown of the most notableissues, categorized by impact:Performance & Async Anti-Patterns• Sequential Requests Defeating Asyncluines ch-0onsioe_collect_ from_es, the code loopsthrough raw_samples and await sasession.post request for each sampleone by one. This means the script waitsfor one HTTP request to finish beforestarting the next, completely defeatingthe purpose of using aiohttp forasynchronous concurrency.• Fix: Collect these coroutines into alist and use asyncio.gather()(ideally wrapped in anasyncio. Semaphore to avoidoverwhelming the endpoint) to runthe replayability probes concurrently.• Blocking File I/O in Async Functions(Lines 110-114, 184-188, 268-269):Standard synchronous file operations(with open(...), json. dump,json.load ) are used inside async def• Enter a prompt for Gemini+Pro vYour Jiminny chats aren't used to improve our models. Gemini is Alane can make mistakes, Inciualne about peodleYour privacy & GeminiSummarize page328329330 +→0 • Daily - Platform • nowA100% CD8 • Tue 14 Apr 9:51:28• 2 1 203 viewed• Awaltind aporoval+330 UUUUu• Viewedparser = create_parser(AiActivityTypeEvaluator)parser.aad arcument"_-data-set",lyoe-sulndetault="initial",dest="data_set",nelo= Vataser Toler uncer eval outvurral acuvity tviereval catasets, etault.initial)",parser.aad arcument"--populate-ground-truth-1lm",action="store_true",dest="populate_ground_truth_llm",parser.add araument"--ground-truth-llm-endpoint",type=str,default="OPENAI: gpt-5_2",dest="ground_truth_llm_endpoint",parser.add_argument("--compare-to-ground-truth",type=str,choices=["es", "1lm"],default="es",dest="compare_to_ground_truth",aros = parser.parse arasAiActivityTypeEvaluator.DATA_SET = args.data_setA1Activitylypeevaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_Lm)AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpointAiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truthasyncio.run(run_evaluation(AiActivityTypeEvaluator, args))> .gitignore & ++4 0000• Viewed...You made it to the end!...
|
NULL
|
NULL
|
NULL
|
8548
|
|
DiaFilelEditViewTabsBookmarksHistoryExtensionsWind DiaFilelEditViewTabsBookmarksHistoryExtensionsWindowHelpmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)M inbox (2) - nii22 Pluctorm SprlC 3y 19798 eva(l Service-DeskJr Foet AmesonxCost | Claude@ Actions secrC C Dependathttps://jiminny.atlassian.net/jira/software/c/projects/JY/boards/37• Sminny0 Projectslã AWsE SSHElDatadog * Claude5 CircieclA Sentry|8 TasksPlatform TeamQ Search board8800018EpicvTypevLabel vQuick filtersREADY FOR DEV1INOEV 4COOE REVIEW 3BLOCKED[Part? Automated reoortsexceed Fontawesomepackage limitsMASNTENANCISReady for Dev-205641П ****=AJ PEPORTSIn DevBE4-18400 115n****= 1Delays in CRM SyncCLATFORM STARUTTYIn DevQJy-20553workfiow for automatedDependabot fixesEMPROVEMENT OF OUR FIFFICHINGYIn Dev₴ J-2055120000Sru and sboize wodllelOS using now mobile ose...MASNTENANCSIn DevQAJy-20431Evaluation for AI ActivityTypesAUTO-OETICTED AETIVITY TYPGCode Review+ Create-J-Wies5 -•00=AJ Reports TrackingCode Review0 J-20543 | 11 •0• =for SSO for 'persistent'name,id.formatREDUCE CHURNCode ReviewД -л1-2063205 •*=+ Create9:51 AMDaily - Platform(ah)Daily - Platform - now8•• Actions sec:( Dependabot-Insights & Coachin.• DevDux.Compiete sprintPO ACCEPTANCEO Far Dependa:D Al BookmarksGrouo: QueriesDEPLOYHx oowntnte ouring teteature flag reieaseBUPPORT TICKETS0 -20585 15 + ****=AI Review - Q1|CTHEMATAOOCOeployed• -19982 1 @ 00.0=antity "Event"PLATTORM STARUITYDeployed© _20384 08 1 •*•*=Missed deletion of stale0 -20619 1 11 •..•:(Pipedrive) DI > status notlabeled, date data missingCEOUCT CKOR0 -200м 05 [1 ****=Steika StoyanovaSteliyan GeorgievKovalik6:068 • Tue 14 Apr 9:51:30ChatNikolay Yankov*Nikolay lvanov...
|
NULL
|
NULL
|
NULL
|
8549
|
|
FirefoxFileEoitViewHistoryBookmarksProfilesToolsWi FirefoxFileEoitViewHistoryBookmarksProfilesToolsWindow Helpgithub.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8~ Google Gemini...l1 OpenJy 19798 evaluation for ai activity types #468All commitsPlatform Sprint 1 Q2 - Platform Tea@ Jy 19798 evaluation for ai activ+ New labnikolaybiaivanov wants to merge 9 commits into master from JY-19798-evaluation-for-ai-activity-types [v src/eval/eval-ai-activity-type.py SQ Filter files...• E eval_output/ai_activity_type/eval…..> • ground_truth>I inoutE src/eval* endpoints.pyE eval-ai-activity-type.py#.gitignore293294299are there soem issueswith this PRShow thinking vYes, there are several issues and code smellsin this Pull Request, ranging from performancebottlenecks to architectural anti-patterns.Here is a breakdown of the most notableissues, categorized by impact:Performance & Async Anti-Patterns• Sequential Requests Defeating Asyncluines ch-06t nsiee_collect_from_es, the code loopsthrough raw_samples and await sasession.post request for each sampleone by one. This means the script waitsfor one HTTP request to finish beforestarting the next, completely defeatingthe purpose of using aiohttp forasynchronous concurrency.• Fix: Collect these coroutines into alist and use asyncio.gather()(ideally wrapped in anasyncio. Semaphore to avoidoverwhelming the endpoint) to runthe replayability probes concurrently.• Blocking File I/O in Async Functions(Lines 110-114, 184-188, 268-269):Standard synchronous file operations(with open(...), json. dump,json.load ) are used inside async def• Enter a prompt for Gemini+Pro vYour Jiminny chats aren't used to improve our models. Gemini is Alane can make mistakes, Inciualne aoout peodleYour privacy & GeminiSummarize page328329330 +‹-0ll • | Daily - Platform • nowA100% CD8 • Tue 14 Apr 9:51:31• 2 1 203 viewed•Awaltind aporova+330 UUUUu• Viewedparser = create_parser(AiActivityTypeEvaluator)parser.aad arcument"_-data-set",lyoe-sutndetault="initial",dest="data_set",nelo= Vataser Toler uncer eval outvurral acuvity tviereval catasets, etault.initial)",parser.aad arcument"--populate-ground-truth-1lm",action="store_true",dest="populate_ground_truth_llm",parser.add araument"--ground-truth-llm-endpoint",type=str,default="OPENAI: gpt-5_2",dest="qround truth llm endpoint".parser.add_argument("--compare-to-ground-truth",type=str,choices=["es", "1lm"],default="es",dest="compare_to_ground_truth",aros = parser.parse arasAiActivityTypeEvaluator.DATA_SET = args.data_setA1Activitylypeevaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_Lm)AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpointAiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truthasyncio.run(run_evaluation(AiActivityTypeEvaluator, args))> .gitignore & ++4 0000• ViewedYou made it to the end!...
|
NULL
|
NULL
|
NULL
|
8550
|
|
DialFilelEditViewTabsBookmarksHistoryExtensionsWin DialFilelEditViewTabsBookmarksHistoryExtensionsWindowHelp• 0.(ah)Daily - Platform - nowmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)8•nbox (2) - nil22 Pluctorm SprlC 3y 19798 eva(l Service-Desk xJr Foet AmesonCost | Claude) Actions secrC C Dependat• Actions sec:https://jiminny.atlassian.net/jira/software/c/projects/JY/boards/377selectedissue=JY-20543Simir0 ProjectsBI AWSE SSHDatados0 • * Claudel3 CircieciSentry8 TasksInsights & Coachin.Platform Team800$ Jy-19240 / [ JY-20543READY FOR DEV 1AJ Reports > TrackinghooesnetxWrexceed Fontawesomepackage limitsMASNTENANEISReady for Dev@-205641 П ****:DescriptionWe want to be able to track the usage of the AJ reports. We will use this to keep track of the adoption but also to useUserpilos tooiops to pust users wno are hot Usino i to use ic• track each generated reporis in bo - include compary name and trequency)• for AJ reports - track each generated report in UserPilot as an event on the user - track it onty for the user who hascreated the report• for Exec reports - track each generated report - set the tracking for each user in the non-jiminny participants list |note: for UP you can see how we currenty track events such as Logged-activity, Held-conferenceSubtasksAdd subtaskLinked work itemsAod unkee work terActivityAdd a comment..Suggest a reply...Status update...ThanksPro tip: press (M to comment~ Details@ Lukas KovallkAswentoeReporter2 Galya DimitrovaComponentsSub-ProductAod opconsLabeisNonelStory point estimateStory PointsOrganisationsWonelPriority( Dependabot-• DelDux.O Far Dependol0 Al BookmarksIZ Improve Story9:51 AMDaily - PlatformGroup: Queriesleature fiag releaseDeployed0 -205e5 15 * ****=AI Review - Q1CTNTHEVNTAORCENDeployed19982[04080-No such column "Type' onentity 'Event'PLATTORM STARTLITYDeployed9 -20384 05 1 ****=Missed deletion of stale0 -40078080(Pipedrive) DI > status notlabeled, date data missingREOUCE CHURNNDeployed0 -20614 05 Пl ****=:Stetka StoyanovaSteliyan GeorgievKovalik6:098 • Tue 14 Apr 9:51:33ChatNikolay YankovNikolay Ivanov...
|
NULL
|
NULL
|
NULL
|
8551
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Zoom in
Open in new window
Enter Full Screen
Stefka Stoyanova
Nikolay Yankov
Steliyan Georgiev
Nikolay Ivanov
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:51
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8552
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Zoom in
Open in new window
Enter Full Screen
Stefka Stoyanova
Nikolay Yankov
Steliyan Georgiev
Nikolay Ivanov
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:51
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8553
|
|
DiaFileEditViewTabsBookmarksHistoryExtensionsWindo DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelp(ablDaily - Platform • now100% <78• Tue 14 Apr 9:51:35meet.google.com / Daily - PlatformNikolay Yankov (Presenting)Chat8•nbox (2) - nilPlatform TeamREADY FOR DEV 1oohesnetixWinexceed Fontawesomepackage limitsMASNTENANCSReady for Dev8-20564122 Pluctorm SprlC 3y 19798 eva(l Service-Desk xIr Foet AwesonCost | Claude) Actions secrC C Dependat• Actions sec:https://jiminny.atlassian.net/jira/software/c/projects/JY/boards/377selectedissue=JY-205430 Projects# AWS|E SSHEl Datadog* Claude3 CircieciSentryD TasksInsights & Coachin.800$ Jy-19240 / A JY-20543AJ Reports > TrackingDescriptionWe want to be able to track the usage of the AJ reports. We will use this to keep track of the adoption but also to useUserpilot tooltips to push users who are not using it to use it.• track each generated reports in DD - Include company name and frequency• for AJ reports - track each generated report in UserPilot as an event on the user - track it only for the user who hascreated the report• for Exec reports - track each generated report - set the tracking for each user in the non-jiminny participants listnote: for UP you can see how we currently track events such as Logged-activity, Held-conferenceSubtasksnousuueskLinked work itemsAos unkee wotk tenActivityAod a commentt.Suggest a reply…Status update…..CooeKowwyDetails@ Lukas KovallkReporter2 Galya DimitrovaUeyele enene@ Open with VS Code|1 branch4 commits1 pull request2 buildsComponentsPlatformSuoerooueAdd optionsNoneStory point estimate( Dependabot-• DevDux.O Far DependolD Al BookmarksXx Improve Story5 days agcUAEh9:51 AMDaily - PlatformGroup: Queriesleature fiag releaseDeployed0 -205e5 15 + ****=AI Review - Q1CNIHEVAORCEVIDeployed_19982 104000-No such column "Type' onentty 'Event"PLATTORM STABTLITYDeployed2038405 0800=Missed deletion of stale0 m40n78080(Pipedrive) DI > status notlabeled, date data missingDeployed0 -20614 05 П ****=:Stefka StoyanovaNikolay YankovSteliyan GeorgievNikolay IvanovLukas Kovalik6:11...
|
NULL
|
NULL
|
NULL
|
8554
|
|
Platform Sprint 1 Q2 - Platform Tea@ Jy 19798 eval Platform Sprint 1 Q2 - Platform Tea@ Jy 19798 evaluation for ai activ+ New labHistoryBookmarksProfilesToolsWindow Helpgithub.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8~ Google Gemini...soccce Cdcsediie3.Run: Executes the evaluation viaasyncio.run(run_evaluation(...)),which orchestrates the collection,endpoint calling, and result scoringolcses.are there soem issueswith this PRShow thinking vYes, there are several issues and code smellsin this Pull Request, ranging from performancebottlenecks to architectural anti-patterns.Here is a breakdown of the most notableissues, categorized by impact:Performance & Async Anti-Patterns• Sequential Requests Defeating Async(Lines 86-106): Inside_collect_from_es, the code loopsthrough raw_samples and await sasession.post request for each sampleone by one. This means the script waitsfor one HTTP request to finish beforestarting the next, completely defeatingthe purpose of using aiohttp forasynchronous concurrency.• Fix: Collect these coroutines into alist and use asyncio.gather ()(ideally wrapped in anasyncio.Semaphore to avoidoverwhelming the endpoint) to run• Enter a prompt for GeminiPro vl1 OpenJy 19798 evaluation for ai activity types #468All commitsnikolaybiaivanov wants to merge 9 commits into master from JY-19798-evaluation-for-ai-activity-types [v src/eval/eval-ai-activity-type.py SQ Filter files...• E eval_output/ai_activity_type/eval…..> • ground_truth>I inoutE src/eval= enapolnts.pyE eval-ai-activity-type.pyE .gitignoreYour Jiminny chats aren't used to improve our models. Gemini is Alane can make mistakes, Inciualne aoout peodleYour privacy & GeminiSummarize page>.gitignore ?293299+329330< 40 lhl | Daily - Platform • now100% [42' 8• Tue 14 Apr 9:51:36• 2 1 203 viewed• Awallind aporovall+330 UUUUU |Viewednarser = create narser AiActivitvTvoervaluator)parser.add_argument("--data-set",lyoe-sutndefault="initial",dest="data_set",helo="Dataset folder under eval outout/ai activity tvoe/eval datasets/ (default:initial)",parser.add_argument("--populate-ground-truth-Ulm",action="store_true",dest="populate_ground_truth_llm",parser.add_argument("_-ground-truth-llm-endpoint",nvoessur.default="OPENAI: gpt-5_2",dest="ground_truth_llm_endpoint",parser.add_argument("--compare-to-ground-truth",type=str,choices=["es", "1lm"],derault=es,dest="compare_to_ground_truth",aros = parser.parse arasAiActivityTypeEvaluator.DATA_SET = args.data_setA1Activitylypeevaluator.POPULATE_GROUNd_TRUTH_LLM = bool(args.populate_ground_truth_(m)AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpointAiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truthasyncio.run(run_evaluation(AiActivityTypeEvaluator, args))+4 0000• Viewed••You made it to the end!...
|
NULL
|
NULL
|
NULL
|
8555
|
|
Platform Sprint 1 Q2 - Platform Tea@ Jy 19798 eval Platform Sprint 1 Q2 - Platform Tea@ Jy 19798 evaluation for ai activ+ New labHistoryBookmarksProfilesToolsWindow Helpgithub.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8~ Google Gemini...soccce Cdcsediie3.Run: Executes the evaluation viaasyncio.run(run_evaluation(...)),which orchestrates the collection,endpoint calling, and result scoringoeses.are there soem issueswith this PRShow thinking vYes, there are several issues and code smellsin this Pull Request, ranging from performancebottlenecks to architectural anti-patterns.Here is a breakdown of the most notableissues, categorized by impact:Performance & Async Anti-Patterns• Sequential Requests Defeating Async(Lines 86-106): Inside_collect_from_es, the code loopsthrough raw_samples and await sasession.post request for each sampleone by one. This means the script waitsfor one HTTP request to finish beforestarting the next, completely defeatingthe purpose of using aiohttp forasynchronous concurrency.• Fix: Collect these coroutines into alist and use asyncio.gather ()(ideally wrapped in anasyncio. Semaphore to avoidoverwhelming the endpoint) to run• Enter a prompt for GeminiPro vl1 OpenJy 19798 evaluation for ai activity types #468All commitsnikolaybiaivanov wants to merge 9 commits into master from JY-19798-evaluation-for-ai-activity-types [v src/eval/eval-ai-activity-type.py SQ Filter files...• E eval_output/ai_activity_type/eval…..> • ground_truth>I inoutE src/eval= enapolnts.pyE eval-ai-activity-type.pyE .gitignoreYour Jiminny chats aren't used to improve our models. Gemini is Alane can make mistakes, Inciualne aoout peodleYour privacy & GeminiSummarize page>.gitignore ?293294299+329330< 40 lhl | Daily - Platform • now100% [42' 8• Tue 14 Apr 9:51:37• 2 1 203 viewed•Awallind aporovall+330 UUUUU |Viewednarser = create narser AiActivitvTvoervaluator)parser.add_argument("--data-set",lyoe-sulndefault="initial,dest="data_set",helo="Dataset folder under eval outout/ai activity tvoe/eval datasets/ (default:initial)",parser.add_argument("--populate-ground-truth-Ulm",action="store_true",dest="populate_ground_truth_llm",parser.add_argument("_-ground-truth-llm-endpoint",nvoessur.default="OPENAI: gpt-5_2",dest="ground_truth_llm_endpoint",parser.add_argument("--compare-to-ground-truth",type=str,choices=["es", "1lm"],derault=es,dest="compare_to_ground_truth",aros = parser.parse arasAiActivityTypeEvaluator.DATA_SET = args.data_setA1Activitylypeevaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_(m)AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpointAiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truthasyncio.run(run_evaluation(AiActivityTypeEvaluator, args))+4 0000• Viewed••You made it to the end!...
|
NULL
|
NULL
|
NULL
|
8556
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
You can't unmute someone else
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:51
AM
Daily - Platform
Daily - Platform
Audio settings
Turn off microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is on.
meet.google.com / Daily - Platform
DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelp(ablDaily - Platform • now100% <478• Tue 14 Apr 9:51:39meet.google.com / Daily - PlatformNikolay Yankov (Presenting)Chat8•nbox (2) - nilPlatform TeamREADY FOR DEV 1oohesnetixWinexceed Fontawesomepackage limitsMASNTENANCSReady for Dev@-205641 П ****:22 Pluctorm SprlC 3y 19798 eva(l Service-Desk xIr Foet AwesonCost | Claude) Actions secrC C Dependat• Actions sec:https://jiminny.atlassian.net/jira/software/c/projects/JY/boards/377selectedissue=JY-205430 Projects# AWS|E3 SSHEl Datadog* Claude3 CircieciSentryD TasksInsights & Coachin.800$ Jy-19240 / A JY-20543AJ Reports > TrackingDescriptionWe want to be able to track the usage of the AJ reports. We will use this to keep track of the adoption but also to useUserpilot tooltips to push users who are not using it to use it.• track each generated reports in DD - Include company name and frequency• for AJ reports - track each generated report in UserPilot as an event on the user - track it only for the user who hascreated the report• for Exec reports - track each generated report - set the tracking for each user in the non-jiminny participants listnote: for UP you can see how we currently track events such as Logged-activity, Held-conferenceSubtasksnousuueskLinked work itemsAos unkee wotk tenActivityAod a commentt.Suggest a reply…Status update...CooKowewyDetails@ Lukas KovallkReporter2 Galya DimitrovaUeyele enene@ Open with VS Code|1 branch4 commits1 pull request2 buildsComponentsPlatformSuoerooueAdd optionsNoneStory point estimate( Dependabot-• DevDux.O Far DependolD Al BookmarksXx Improve Story5 days agcUAEhGroup: Queriesleature fiag reieaseDeployed0 -205e5 15 + ****=•AI Review - Q1CTNIHEVAORCEVIDeployed19982 1 @ ....=No such column "Type' onentty 'Event"PLATTORM STARTLITYDeployed2038405 0800=Missed deletion of stale0-40003 409000(Pipedrive) DI > status notlabeled, date data missingDeployed0 -20614 05 П ****=Stefka StoyanovalNikolay YankovSteliyan GeorgievNikolay IvanovKovalik9:51 AM | Daily - Platform:6:15...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8557
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
You can't unmute someone else
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:52
AM
Daily - Platform
Daily - Platform
Audio settings
Turn off microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is on.
meet.google.com / Daily - Platform
Platform Sprint 1 Q2 - Platform Tea@ Jy 19798 evaluation for ai activ+ New labHistoryBookmarksProfilesToolsWindow Helpgithub.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8~ Google Gemini...soccce Cdcsediie3.Run: Executes the evaluation viaasyncio.run(run_evaluation(...)),which orchestrates the collection,endpoint calling, and result scoringoeses.are there soem issueswith this PRShow thinking vYes, there are several issues and code smellsin this Pull Request, ranging from performancebottlenecks to architectural anti-patterns.Here is a breakdown of the most notableissues, categorized by impact:Performance & Async Anti-Patterns• Sequential Requests Defeating Async(Lines 86-106): Inside_collect_from_es, the code loopsthrough raw_samples and await sasession.post request for each sampleone by one. This means the script waitsfor one HTTP request to finish beforestarting the next, completely defeatingthe purpose of using aiohttp forasynchronous concurrency.• Fix: Collect these coroutines into alist and use asyncio.gather ()(ideally wrapped in anasyncio. Semaphore to avoidoverwhelming the endpoint) to run• Enter a prompt for GeminiPro vl1 OpenJy 19798 evaluation for ai activity types #468All commitsnikolaybiaivanov wants to merge 9 commits into master from JY-19798-evaluation-for-ai-activity-types [v src/eval/eval-ai-activity-type.py SQ Filter files...• E eval_output/ai_activity_type/eval…..> • ground_truth>I inoutE src/eval= enapolnts.pyE eval-ai-activity-type.pyE .gitignoreYour Jiminny chats aren't used to improve our models. Gemini is Alane can make mistakes, Inciualne aoout peodleYour privacy & GeminiSummarize page>.gitignore ?293299+329330< 40 lhl | Daily - Platform • now100% [42' 8• Tue 14 Apr 9:52:07• 2 / 203 viewed•Awallind aporovall+330 UUUUU |Viewednarser = create narser AiActivitvTvoervaluator)parser.add_argument("--data-set",lyoe-sulndefault="initial,dest="data_set",helo="Dataset folder under eval outout/ai activity tvoe/eval datasets/ (default:initial)",parser.add_argument("--populate-ground-truth-Ulm",action="store_true",dest="populate_ground_truth_llm",parser.add_argument("_-ground-truth-llm-endpoint",nvoessur.default="OPENAI: gpt-5_2",dest="ground_truth_llm_endpoint",parser.add_argument("--compare-to-ground-truth",type=str,choices=["es", "1lm"],derault=es,dest="compare_to_ground_truth",aros = parser.parse arasAiActivityTypeEvaluator.DATA_SET = args.data_setA1Activitylypeevaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_(m)AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpointAiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truthasyncio.run(run_evaluation(AiActivityTypeEvaluator, args))+4 0000• Viewed••You made it to the end!...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8558
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
You can't unmute someone else
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:52
AM
Daily - Platform
Daily - Platform
Audio settings
Turn off microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is on.
meet.google.com / Daily - Platform
DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelp(ablDaily - Platform • nowmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)100% <47•8• Tue 14 Apr 9:52:09Chat68•nbox (2) - nil22 Pluctorm SprlC 3y 19798 eva(l Service-Desk xIr Foet AwesonCost | Claude) Actions secrC C Dependat• Actions sec:( Dependabot-https://jiminny.atlassian.net/jira/software/c/projects/JY/boards/377selectedissue=JY-205430 ProjectsB AWSE3 SSHEl Datadog* Claude3 CircieciSentryD TasksInsights & Coachin.• DevPlatform Team800$ Jy-19240 / A JY-20543READY FOR DEV 1AJ Reports > TrackingoohesnetixWinexceed Fontawesomepackage limitsMASNTENANCSReady for Dev8-205641DescriptionWe want to be able to track the usage of the AJ reports. We will use this to keep track of the adoption but also to useUserpilot tooltips to push users who are not using it to use it.• track each generated reports in DD - Include company name and frequency• for AJ reports - track each generated report in UserPilot as an event on the user - track it only for the user who hascreated the report• for Exec reports - track each generated report - set the tracking for each user in the non-jiminny participants listnote: for UP you can see how we currently track events such as Logged-activity, Held-conferenceSubtasksnousuueskLinked work itemsAos unkee wotk tenActivityAod a commentt.Suggest a reply…Status update...Dux.CooeKowwyXx Improve StoryDetails@ Lukas KovallkReporter2 Galya DimitrovaUeyele enene@ Open with VS Code|1 branch4 commits1 pull request2 builds5 days agcUAEhComponentsPlatformSuoerooueAdd optionsNoneStory point estimateO Far DependolD Al BookmarksGroup: Queriesleature fiag releaseDeployed0 -20585 15 * ****=AI Review - Q1CTNIHEVAORCEVIDeployed_19982No such column "Type' onentty 'Event"PLATTORM STARTLITYDeployed2038405 0000=Missed deletion of stale0 m400(Pipedrive) DI > status notlabeled, date data missingDeployed0 -20614 05 П ****=Stefka StoyanovaNikolay YankovSteliyan Georgiev,Nikolay Ivanovfalik9:52 AM Daily - Platform:6:45...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8559
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
You can't unmute someone else
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:52
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform
DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelp(ablDaily - Platform • nowmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)100% <78• Tue 14 Apr 9:52:13Chat68•nbox (2) - nil22 Pluctorm SprlC 3y 19798 eva(l Service-Desk xIr Foet AwesonCost | Claude) Actions secrC C Dependat• Actions sec:( Dependabot-https://jiminny.atlassian.net/jira/software/c/projects/JY/boards/377selectedissue=JY-205430 ProjectsB AWSE3 SSHEl Datadog* Claude3 CircieciSentryD TasksInsights & Coachin.• DevPlatform Team800$ Jy-19240 / A JY-20543READY FOR DEV 1AJ Reports > TrackingoohesnetixWinexceed Fontawesomepackage limitsMASNTENANCSReady for Dev8-205641DescriptionWe want to be able to track the usage of the AJ reports. We will use this to keep track of the adoption but also to useUserpilot tooltips to push users who are not using it to use it.• track each generated reports in DD - Include company name and frequency• for AJ reports - track each generated report in UserPilot as an event on the user - track it only for the user who hascreated the report• for Exec reports - track each generated report - set the tracking for each user in the non-jiminny participants listnote: for UP you can see how we currently track events such as Logged-activity, Held-conferenceSubtasksnousuoes%Linked work itemsAos unkee wotk tenActivityAod a commentt.Suggest a reply…Status update...Uro bpc press M lo clDux.CooKowewyXx Improve StoryDetails@ Lukas KovallkReporter2 Galya DimitrovaUeyele enene@ Open with VS Code|1 branch4 commits1 pull request2 builds5 days agcUAEhComponentsPlatformSuoerooueAdd optionsNoneStory point estimateO Far DependolD Al BookmarksGroup: Queriesleature fiag reieaseDeployed0 -205e5 15 + ****=AI Review - Q1CTNIHEVAORCEVIDeployed_19982No such column "Type' onentty 'Event"PLATTORM STABTLITYDeployed2038405 0000=Missed deletion of stale0 m400(Pipedrive) DI > status notlabeled, date data missingDeployed0 -20614 05 П ****=Stefka StoyanovaNikolay YankovSteliyan GeorgievNikolay Ivanovralik9:52 AM Daily - Platform:6:49...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8560
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
You can't unmute someone else
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:52
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform
Platform Sprint 1 Q2 - Platform Tea@ Jy 19798 evaluation for ai activ+ New labHistoryBookmarksProfilesToolsWindow Helpgithub.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8~ Google Gemini...soccce Cdcsediie3.Run: Executes the evaluation viaasyncio.run(run_evaluation(...)),which orchestrates the collection,endpoint calling, and result scoringoeses.are there soem issueswith this PRShow thinking vYes, there are several issues and code smellsin this Pull Request, ranging from performancebottlenecks to architectural anti-patterns.Here is a breakdown of the most notableissues, categorized by impact:Performance & Async Anti-Patterns• Sequential Requests Defeating Async(Lines 86-106): Inside_collect_from_es, the code loopsthrough raw_samples and await sasession.post request for each sampleone by one. This means the script waitsfor one HTTP request to finish beforestarting the next, completely defeatingthe purpose of using aiohttp forasynchronous concurrency.• Fix: Collect these coroutines into alist and use asyncio.gather ()(ideally wrapped in anasyncio. Semaphore to avoidoverwhelming the endpoint) to run• Enter a prompt for GeminiPro vl1 OpenJy 19798 evaluation for ai activity types #468All commitsnikolaybiaivanov wants to merge 9 commits into master from JY-19798-evaluation-for-ai-activity-types [v src/eval/eval-ai-activity-type.py SQ Filter files...• E eval_output/ai_activity_type/eval…..> • ground_truth>I inoutE src/eval= enapolnts.pyE eval-ai-activity-type.pyE .gitignoreYour Jiminny chats aren't used to improve our models. Gemini is Alane can make mistakes, Inciualne aoout peodleYour privacy & GeminiSummarize page>.gitignore ?293299+329330< 40 lhl | Daily - Platform • now100% [42' 8• Tue 14 Apr 9:52:13• 2 / 203 viewed•Awallind aporovall+330 UUUUU |Viewednarser = create narser AiActivitvTvoervaluator)parser.add_argument("--data-set",lyoe-sulndefault="initial,dest="data_set",helo="Dataset folder under eval outout/ai activity tvoe/eval datasets/ (default:initial)",parser.add_argument("--populate-ground-truth-Ulm",action="store_true",dest="populate_ground_truth_llm",parser.add_argument("_-ground-truth-llm-endpoint",nvoessur.default="OPENAI: gpt-5_2",dest="ground_truth_llm_endpoint",parser.add_argument("--compare-to-ground-truth",type=str,choices=["es", "1lm"],derault=es,dest="compare_to_ground_truth",aros = parser.parse arasAiActivityTypeEvaluator.DATA_SET = args.data_setA1Activitylypeevaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_(m)AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpointAiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truthasyncio.run(run_evaluation(AiActivityTypeEvaluator, args))+4 0000• Viewed••You made it to the end!...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8561
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
You can't unmute someone else
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:52
AM
Daily - Platform
Daily - Platform
Audio settings
Turn off microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is on.
meet.google.com / Daily - Platform
DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelp(ablDaily - Platform • now100% [8• Tue 14 Apr 9:52:18meet.google.com / Daily - PlatformNikolay Yankov (Presenting)Chat8•nbox (2) - nilPlatform TeamQ Search boardREADY FOR DEV 1oohesnetixWinexceed Fontawesomepackage limitsMASNTENANCSReady for Dev@-205641 П ****:22 Pluctorm SprlC 3y 19798 eva(l Service-Desk xIr Foet AwesonCost | Claude) Actions secrC C Dependathttps://jiminny.atlassian.net/jira/software/c/projects/JY/boards/377selectedissue=JY-205430 Projects# AWS|E SSHE Datados0 • * Ciaudel3 CircieciSentry8 Tasks800$ Jy-19240 / A JY-20543AJ Reports > TrackingDescriptionWe want to be able to track the usage of the AJ reports. We will use this to keep track of the adoption but also to useUserpilos tooiops to pusn users wno are hot usino i to use ic• track each generated reports in DD - Include company name and frequency• for AJ reports - track each generated report in UserPilot as an event on the user - track it only for the user who hascreated the report• for Exec reports - track each generated report - set the tracking for each user in the non-jiminny participants listnote: for UP you can see how we currently track events such as Logged-activity, Held-conferenceSubtasksnousuoes%Linked work itemsAos unkee wotk tenActivityAoda commenttSuggest a reply…Status update...Uro bpc press M loo• Actions sec:Insights & Coachin.( Dependabot-• DevDux.O Far DependolD Al BookmarksCooeKowewyDetails@ Lukas KovallkReporter2 Galya DimitrovaDevelopment #@ Open with VS Code|1 branch4 commits1 pull request2 buildsComponentsPlatformSuoerooueAdd optionsNoneStory point estimateXx Improve Story5 days agcOMEN C +Group: Queriesleature fiag releaseDeployed0 -205e5 15 + ****=•AI Review - Q1CTNTHEVANAORCEVDeployed19982wo sucn coumh "lype onentty 'Event"PLATTORM STARTLITYDeployed2038405 0000=Missed deletion of stale0 m400(Pipedrive) DI > status notlabeled, date data missingDeployed0 -20614 05 П ****=Stefka StoyanovaNikolay YankovSteliyan GeorgievNikolay IvanovValik9:52 AM Daily - Platform:6:54...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8562
|
|
DiaFileEditViewTabsBookmarksHistoryExtensionsWindo DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelp(ablDaily - Platform • nowmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)100% [8• Tue 14 Apr 9:52:40Chat68•Platform TeamREADY FOR DEV 1doesnetixWiwexceed Fontawesomepackage limitsMAENTENANCISReady for Dev8-20564 1 П ****=G 3y 19798 eva(l Service-Desk xJk Foet Aweson• Cost | Claude) Actions secrC C Dependat•) Actions secr( Dependabot-sny.atlassian.net/jira/software/c/projects/JY/boards/37?selectedissue=JY-205430 Projects# AWS|E SSHE Datados0 C * Claudel3 Circieci[ Sentry8 Tasks• Del800$ Jy-19240 / A JY-20543AJ Reports > Tracking~ DescriptionWe want to be able to track the usage of the AJ reports. We will use this to keep track of the adoption but also to useUserpilos tooiops to pusn users wno are not usino i to use ic• track each generated reports in DD - Include company name and frequency• for AJ reports - track each generated report in UserPilot as an event on the user - track it only for the user who hascreated the report• for Exec reports - track each generated report - set the tracking for each user in the non-jiminny participants list |note: for UP you can see how we currenty track events such as Logged-activity, Held-conferenceSubtasksnousuueskLinked work itemsAod unkee work erActivityAoda commenttSuggest a reply.Status update...Uro bpc press M looInsights & Coachin.D ux.@21Cooe KowwwyXZ Improve Story~ Details@ Lukas KovallkReporter2 Galya DimitrovaUerele enene@ Open with VS Code|1 branch4 commits1 pull request2 builds5 days agcUrEhComponentsPlatformSuoerooueAdd optionsNoneStory point estimateO Far DependolD Al Bookmarks9:52 AM Daily - PlatformGroup: QueriesDEPLOY &Puooanonoeleature fiag reieaseDeployed120065 15 * *000=AI Review - Q1CTNIHEWANAORCOVDeployed_19982o 4080-wo sucn coumh "lype onentity 'Event"PLATTORM STARTLITYDeployed2038405 0600=Missed deletion of stale0 -400R408009(Pipedrive) DI > status notlabeled, date data missingREDUCE CHURNNDeployed0 -20014 05 П ****=:Stefka StoyanovaNikolay YankovSteliyan GeorgievNikolay Ivanovovalik• 7:16...
|
NULL
|
NULL
|
NULL
|
8563
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
You can't unmute someone else
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:52
AM
Daily - Platform
Daily - Platform
Audio settings
Turn off microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is on.
meet.google.com / Daily - Platform
Platform Sprint 1 Q2 - Platform Tea@ Jy 19798 evaluation for ai activ+ New labHistoryBookmarksProfilesToolsWindow Helpgithub.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8~ Google Gemini...soccce Cdcsediie3.Run: Executes the evaluation viaasyncio.run(run_evaluation(...)),which orchestrates the collection,endpoint calling, and result scoringoeses.are there soem issueswith this PRShow thinking vYes, there are several issues and code smellsin this Pull Request, ranging from performancebottlenecks to architectural anti-patterns.Here is a breakdown of the most notableissues, categorized by impact:Performance & Async Anti-Patterns• Sequential Requests Defeating Async(Lines 86-106): Inside_collect_from_es, the code loopsthrough raw_samples and await sasession.post request for each sampleone by one. This means the script waitsfor one HTTP request to finish beforestarting the next, completely defeatingthe purpose of using aiohttp forasynchronous concurrency.• Fix: Collect these coroutines into alist and use asyncio.gather ()(ideally wrapped in anasyncio. Semaphore to avoidoverwhelming the endpoint) to run• Enter a prompt for GeminiPro vl1 OpenJy 19798 evaluation for ai activity types #468All commitsnikolaybiaivanov wants to merge 9 commits into master from JY-19798-evaluation-for-ai-activity-types [v src/eval/eval-ai-activity-type.py SQ Filter files...• E eval_output/ai_activity_type/eval…..> • ground_truth>I inoutE src/eval= enapolnts.pyE eval-ai-activity-type.pyE .gitignoreYour Jiminny chats aren't used to improve our models. Gemini is Alane can make mistakes, Inciualne aoout peodleYour privacy & GeminiSummarize page>.gitignore ?293299+329330< 40 lhl | Daily - Platform • now100% (' • Tue 14 Apr 9:52:40• 2 / 203 viewed•Awallind aporovali+330 UUUUU |Viewednarser = create narser AiActivitvTvoervaluator)parser.add_argument("--data-set",lyoe-sulndefault="initial,dest="data_set",helo="Dataset folder under eval outout/ai activity tvoe/eval datasets/ (default:initial)",parser.add_argument("--populate-ground-truth-Ulm",action="store_true",dest="populate_ground_truth_llm",parser.add_argument("_-ground-truth-llm-endpoint",nvoessur.default="OPENAI: gpt-5_2",dest="ground_truth_llm_endpoint",parser.add_argument("--compare-to-ground-truth",type=str,choices=["es", "1lm"],derault=es,dest="compare_to_ground_truth",aros = parser.parse arasAiActivityTypeEvaluator.DATA_SET = args.data_setA1Activitylypeevaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_(m)AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpointAiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truthasyncio.run(run_evaluation(AiActivityTypeEvaluator, args))+4 0000• Viewed••You made it to the end!...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8564
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
You can't unmute someone else
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:52
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform
DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelp(ablDaily - Platform • nowmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)100% <478• Tue 14 Apr 9:52:47Chat68•Platform TeamREADY FOR DEV 1doesnetixWiwexceed Fontawesomepackage limitsMAENTENANCISReady for Dev8-205641 П ****=G 3y 19798 eva(l Service-Desk xJk Foet Aweson• Cost | Claude) Actions secrC C Dependat•) Actions secr( Dependabot: xO Far Dependolsny.atlassian.net/jira/software/c/projects/JY/boards/37?selectedissue=JY-205430 ProjectsBl AWSE3 SSHE Datados0 C * Claudel3 Circieci[ Sentry8 Tasks• Del800$ Jy-19240 / A JY-20543AJ Reports > Tracking~ DescriptionWe want to be able to track the usage of the AJ reports. We will use this to keep track of the adoption but also to useUserpilos tooiops to pusn users wno are not usino i to use ic• track each generated reports in DD - Include company name and frequency• for AJ reports - track each generated report in UserPilot as an event on the user - track it only for the user who hascreated the report• for Exec reports - track each generated report - set the tracking for each user in the non-jiminny participants list |note: for UP you can see how we currenty track events such as Logged-activity, Held-conferenceSubtasksnousuueskLinked work itemsAod unkee work erActivityAoda commenttSuggest a reply.Status update...Uro bpc press M looInsights & Coachin.D ux.@21Cooe KowwwyXZ Improve Story~ Details@ Lukas KovallkReporter2 Galya DimitrovaUerele enene@ Open with VS Code|1 branch4 commits1 pull request2 builds5 days agcUrEhComponentsPlatformSuoerooueAdd optionsNoneStory point estimateD Al Bookmarks9:52 AM Daily - PlatformGroup: QueriesDEPLOY &Puooanonoeleature fiag reieaseDeployed0 -20505 15 + ****=•AI Review - Q1CTNIHEWANAORCOVDeployed_19982o e00=wo sucn co umh "lype onentty 'Event"PLATTORM STARTUITYDeployed2038405 0600=Missed deletion of stale0 -400R408009(Pipedrive) DI > status notlabeled, date data missingDeployed0 -20614 05 П ****=:Stefka StoyanovaNikolay YankovSteliyan GeorgievNikolay Ivanovvalik•7:23...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8565
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
You can't unmute someone else
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:52
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform
Platform Sprint 1 Q2 - Platform Tea@ Jy 19798 evaluation for ai activ+ New labHistoryBookmarksProfilesToolsWindow Helpgithub.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8~ Google Gemini...soccce Cdcsediie3.Run: Executes the evaluation viaasyncio.run(run_evaluation(...)),which orchestrates the collection,endpoint calling, and result scoringoeses.are there soem issueswith this PRShow thinking vYes, there are several issues and code smellsin this Pull Request, ranging from performancebottlenecks to architectural anti-patterns.Here is a breakdown of the most notableissues, categorized by impact:Performance & Async Anti-Patterns• Sequential Requests Defeating Async(Lines 86-106): Inside_collect_from_es, the code loopsthrough raw_samples and await sasession.post request for each sampleone by one. This means the script waitsfor one HTTP request to finish beforestarting the next, completely defeatingthe purpose of using aiohttp forasynchronous concurrency.• Fix: Collect these coroutines into alist and use asyncio.gather ()(ideally wrapped in anasyncio. Semaphore to avoidoverwhelming the endpoint) to run• Enter a prompt for GeminiPro vl1 OpenJy 19798 evaluation for ai activity types #468All commitsnikolaybiaivanov wants to merge 9 commits into master from JY-19798-evaluation-for-ai-activity-types [v src/eval/eval-ai-activity-type.py SQ Filter files...• E eval_output/ai_activity_type/eval…..> • ground_truth>I inoutE src/eval= enapolnts.pyE eval-ai-activity-type.pyE .gitignoreYour Jiminny chats aren't used to improve our models. Gemini is Alane can make mistakes, Inciualne aoout peodleYour privacy & GeminiSummarize page>.gitignore ?293299+329330< 40 lhl | Daily - Platform • now100% (' • Tue 14 Apr 9:52:47• 2 1 203 viewed•Awallind aporovali+330 UUUUU |Viewednarser = create narser AiActivitvTvoervaluator)parser.add_argument("--data-set",lyoe-sulndefault="initial,dest="data_set",helo="Dataset folder under eval outout/ai activity tvoe/eval datasets/ (default:initial)",parser.add_argument("--populate-ground-truth-Ulm",action="store_true",dest="populate_ground_truth_llm",parser.add_argument("_-ground-truth-llm-endpoint",nvoessur.default="OPENAI: gpt-5_2",dest="ground_truth_llm_endpoint",parser.add_argument("--compare-to-ground-truth",type=str,choices=["es", "1lm"],derault=es,dest="compare_to_ground_truth",aros = parser.parse arasAiActivityTypeEvaluator.DATA_SET = args.data_setA1Activitylypeevaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_(m)AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpointAiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truthasyncio.run(run_evaluation(AiActivityTypeEvaluator, args))+4 0000• Viewed••You made it to the end!...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8566
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
You can't unmute someone else
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:52
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
meet.google.com / Daily - Platform
DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelp(ablDaily - Platform • now100% <78• Tue 14 Apr 9:52:57meet.google.com / Daily - PlatformNikolay Yankov (Presenting)Chat8•Platform TeamREADY FOR DEV 1doesnetixWiwexceed Fontawesomepackage limitsMAENTENANCISReady for Dev8-205641 П ****=G 3y 19798 eva(l Service-Desk xJk Foet Aweson• Cost | Claude) Actions secrC C Dependat•) Actions secr( Dependabot-sny.atlassian.net/jira/software/c/projects/JY/boards/37?selectedissue=JY-205430 Projects# AWS|E SSHE Datados0 C * Claudel3 Circieci[ Sentry8 Tasks• Del800$ Jy-19240 / A JY-20543AJ Reports > Tracking~ DescriptionWe want to be able to track the usage of the AJ reports. We will use this to keep track of the adoption but also to useUserpilot tooltips to push users who are not using it to use it.• track each generated reports in DD - Include company name and frequency• for AJ reports - track each generated report in UserPilot as an event on the user - track it onty for the user who hascreated the report• for Exec reports - track each generated report - set the tracking for each user in the non-fiminny participants listnote: for UP you can see how we currently track events such as Logged-activity, Held-conferenceSubtasksnousuueskLinked work itemsAod unkee work erActivityAoda commenttSuggest a reply.Status update...Uro bpc press M looInsights & Coachin.D ux.@21Cooe KowwwyXZ Improve Story~ Details@ Lukas KovallkReporter2 Galya DimitrovaUerele enene@ Open with VS Code|1 branch4 commits1 pull request2 builds5 days agcUrEhComponentsPlatformSuoeyrooueAdd optionsNoneStory point estimateO Far DependolD Al BookmarksGroup: QueriesDEPLOY &Puooanonoeleature fiag reieaseDeployed2006515 * *000=AI Review - Q1CTNIHEWANAORCOVDeployed-199821 @ •.=•=wo sucn co umh "lype onentty 'Event"PLATTORM STARTLITYDeployed8 -20384 05 11 ****=Missed deletion of stale0 -400R408009(Pipedrive) DI > status notlabeled, date data missingREDUCE CHURNNDeployed0 -20614 05 П ****=Stefka StoyanovaNikolay YankovSteliyanGeorgievNikolay IvanoyKovalik9:52 AM Daily - Platform:• 7:33...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8567
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
You can't unmute someone else
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:52
AM
Daily - Platform
Daily - Platform
Audio settings
Turn off microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
meet.google.com / Daily - Platform
Platform Sprint 1 Q2 - Platform Tea@ Jy 19798 evaluation for ai activ+ New labHistoryBookmarksProfilesToolsWindow Helpgithub.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8~ Google Gemini...soccce Cdcsediie3.Run: Executes the evaluation viaasyncio.run(run_evaluation(...)),which orchestrates the collection,endpoint calling, and result scoringoeses.are there soem issueswith this PRShow thinking vYes, there are several issues and code smellsin this Pull Request, ranging from performancebottlenecks to architectural anti-patterns.Here is a breakdown of the most notableissues, categorized by impact:Performance & Async Anti-Patterns• Sequential Requests Defeating Async(Lines 86-106): Inside_collect_from_es, the code loopsthrough raw_samples and await sasession.post request for each sampleone by one. This means the script waitsfor one HTTP request to finish beforestarting the next, completely defeatingthe purpose of using aiohttp forasynchronous concurrency.• Fix: Collect these coroutines into alist and use asyncio.gather ()(ideally wrapped in anasyncio. Semaphore to avoidoverwhelming the endpoint) to run• Enter a prompt for GeminiPro vl1 OpenJy 19798 evaluation for ai activity types #468All commitsnikolaybiaivanov wants to merge 9 commits into master from JY-19798-evaluation-for-ai-activity-types [v src/eval/eval-ai-activity-type.py SQ Filter files...• E eval_output/ai_activity_type/eval…..> • ground_truth>I inoutE src/eval= enapolnts.pyE eval-ai-activity-type.pyE .gitignoreYour Jiminny chats aren't used to improve our models. Gemini is Alane can make mistakes, Inciualne aoout peodleYour privacy & GeminiSummarize page>.gitignore ?293299+329330< 40 lhl | Daily - Platform • now100% [42' 8• Tue 14 Apr 9:52:57• 2 / 203 viewed•Awallind aporovall+330 UUUUU |Viewednarser = create narser AiActivitvTvoervaluator)parser.add_argument("--data-set",lyoe-sulndefault="initial,dest="data_set",helo="Dataset folder under eval outout/ai activity tvoe/eval datasets/ (default:initial)",parser.add_argument("--populate-ground-truth-Ulm",action="store_true",dest="populate_ground_truth_llm",parser.add_argument("_-ground-truth-llm-endpoint",nvoessur.default="OPENAI: gpt-5_2",dest="ground_truth_llm_endpoint",parser.add_argument("--compare-to-ground-truth",type=str,choices=["es", "1lm"],derault=es,dest="compare_to_ground_truth",aros = parser.parse arasAiActivityTypeEvaluator.DATA_SET = args.data_setA1Activitylypeevaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_(m)AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpointAiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truthasyncio.run(run_evaluation(AiActivityTypeEvaluator, args))+4 0000• Viewed••You made it to the end!...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8568
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
You can't unmute someone else
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:53
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform
DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelp(abl| Daily - Platform • nowmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)100% [8• Tue 14 Apr 9:53:01Chat68108•Inbox (2) - n22 Plactorm SprlG Jy 19798 eval(l Service-Desk xJr Foet Aweson• Cost | Claud x@ Actions secreC C: Dependat• Actions secr( Dependabot: xhttps://jiminny.atlassian.net/jira/software/c/projects/JY/boards/372selectedissue=JY-20632• Sminn• ProjectsE SSHPlatform TeamQ Search board2800018READY FOR DEV 1INDEV 4Part? Automated reoorts)exceed FontawesomeMASNTENANESReady for Dev8-205641AJREPORTSIn DevBEFEA -18900 115n ****= 3Delays in CRM SyncCUATFORM STABTUITNIn DevQJy-20553workfiow for automatedDependabot fixesEMPROVEMENT OF OUR EFFICEINCYIn Dev©J-205512 00005ru and swaoiize wodieOS using new mobile use...MAINTENANCEIn DevQAД -37-20431El Dacadog © * Claude2 CircleclSentry0 Tasks)Insights & Coachin.• Dev• UXLabel vQuick filtersComplete sprintCOOE REVIEW 3BLOCKEDPO ACCEPTANCEEvaluation for AI ActivityTypesAUTO-DETECTED ACTIVITY TYPECode ReviewJ1-17085 -•00=AJ Reports > TrackingCode ReviewД 3-20543 1for SSO for 'persistent'name_id_format (REDUCE CHURNCode ReviewД лл-206320,5 •.• =+ CreateO Far Dependo:D Al BookmarksGroup: QueriesDEPLOY &nix cownome ouring laleature fiag releaseBUPPORT TICKETSDeployed2036515 •000=AI Review - Q1Oeployed0 -19982|1 • •=*=no sucn coumn "lype onentity "Event"PLATTORM STABTUTTYDeployed-20384 05 П ••*•=|Missed deletion of staleREDUCE CHURN0 1200n(Pipedrive) DI > status not|labeled, date data missingREOUCE CHURNDeployed0 -20614 05 П ****=Stefka StoyanovaNikolay YankovSteliyan GeorgievNikolay IvanovKovalik9:53 AM Daily - Platform• 7:37...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8569
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Zoom in
Open in new window
Enter Full Screen
Stefka Stoyanova
Nikolay Yankov
Steliyan Georgiev
Nikolay Ivanov
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:53
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8570
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Zoom in
Open in new window
Enter Full Screen
Stefka Stoyanova
Nikolay Yankov
Steliyan Georgiev
Nikolay Ivanov
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:53
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8571
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Zoom in
Open in new window
Enter Full Screen
Stefka Stoyanova
Nikolay Yankov
Steliyan Georgiev
Nikolay Ivanov
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:53
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8572
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
You can't unmute someone else
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:53
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform
DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelpmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)<> Codenbox (2)C SRD-677(7 (SRO-67C Jy 19798(7 Service-oir Font. Amr• COMTICH• Actions :https://github.comfjminny/app/pull/11935lã AWsE SSH• Home | SalesforceElDatadog * Claude3 CircieciD Sentryjiminny / app &17 Pull requests 24Actions|CD WidSecurity and qualityk Insights@ SettingsSRD-6779 |JY-20632 | Unable to log in to Sidekick with SSO#11935 •I1 Openyalokin-liminoy, wants to merge 5 commits into naster from SP0-6779-sidekick-s50|@) Conversation (5* Commits5B Checks 3Fiezggunged 3yalokin-jiminny commented 5 days a90 - edited-JIRA: SRD-6779 |JY-20632:Preface:• I have tried to fox the SSO via Sidekick login for client innutable - sami2_tenants.id = 10 with changingnane id foreat from persistent to enailAddress• This is a suggested fallback that we could use if the upper configuration is not correct / does not work.• Try to find the user by email as fallback when name_1d_fornat is persistentHDG Oyalokin-jiminny torce-pushed the SAD-6779-s1dekick-ss0 branch from 3472394 to 832624e 5 days ageyalokin-jiminny commented 5 days.ago(Pclaude9:53 AM Daily - Platform(abl| Daily - Platform • now100% C8• Tue 14 Apr 9:53:40Chat68•С Ct: DepeQ Actions sDependa:• Far Depeс *D TasksInsights & Coachin.Q Type (L to searchD Dev• UxTue 14 Apr 9:538) All Bookmarks)Stefka StoyanovaNikolay Yankov+200 -4 BBBeKeviewers€ Vasil-Jiminmy@ vkulovAl least 1 approving review is required to mergethis pull request.Still in progress? Convert to dratAblneeNo one-assign yourseitLabelsNone yetProjectsNone yetMestonewomhestoDevek snientSuccessfully merging this pull request may closethese issues.None yetan GeorgievNikolay IvanovQLukas Kovalik8:16...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8573
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Zoom in
Open in new window
Enter Full Screen
Stefka Stoyanova
Nikolay Yankov
Steliyan Georgiev
Nikolay Ivanov
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:53
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8574
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Zoom in
Open in new window
Enter Full Screen
Stefka Stoyanova
Nikolay Yankov
Steliyan Georgiev
Nikolay Ivanov
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:53
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8575
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Zoom in
Open in new window
Enter Full Screen
Stefka Stoyanova
Nikolay Yankov
Steliyan Georgiev
Nikolay Ivanov
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:54
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8576
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
Mute Stefka Stoyanova's microphone
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:54
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform
DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelpmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)E:1nbox (2)SRD-677C Jy 19798(7 Service-oir Font. Amr• COMTICHhttps://jiminny.atlassian.net/jira/software/c/projects/JY/boards/372selectedissue=JY-20632Simir0 ProjectsB AWSE3 SSHE Datadog3 CircieciPlatform Team800$ 5y-12250 / [ Jy-20632READY FOR DEV 1Prepare fallback with email for SSO for 'persistent name_id_formathoohesnetixWiwexceed Fontawesomepackage limitsMAINTENANCISReady for Dev©-20564 1 П ****=DescriptionEdit descriptionSubtasksAdd subtaskLinked work items0 SRD-6779 Unable to log in to SidelickIN PROGRESS MActivityWOrK lOAdd a commentCan I get more info.?Status update...Fro bpc press M lo comment)9:54 AM Daily - PlatformActions s(abl| Daily - Platform • now100% <478• Tue 14 Apr 9:54:23Chat8•С Ct: DepeActions sDependol• Far DepeD TasksInsights & Coachin• DevD ux.Code Keviw y# Improve Story~ DetailsNikolay NikolovAssign to meKeoohNikolay NikolovWweoenieQ Open with VS Code}J Create branch12 commitsl1 pull requestouikeComponentsPlatformSub-ProductSidekickLabelsNonelStory point estimateNoneOPEN CGroup: Queriesteature fiag reieaseSUPPORT TICKETSDeployed1 2006515 * •000=AI Review - Q1CNIHEWANAORCEVDeployed_190821o de0=No such column "Type' onentity 'Event"PLATTORM STARTLTTYDeployed8 -20384 05 11 ****=|Missed deletion of stale0 -40003400000(Pipedrive) DI > status notlabeled, date data missingREOUCT CHURNNDeployed0 -20614 05 П ****=:D Al BookmarksStefka StoyanovalNikolay Yankovan GeorgievNikolay IvanovkLukas Kovalik8:58...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8577
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
Mute Stefka Stoyanova's microphone
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:54
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Turn on microphone (⌘ + d)
Your microphone is off.
meet.google.com / Daily - Platform
DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelpmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)E:1nbox (2)SRD-677C Jy 19798(7 Service-oir Font. Amr• COMTICHhttps://jiminny.atlassian.net/jira/software/c/projects/JY/boards/372selectedissue=JY-20632Simir0 Projects# AWS|E SSHE Datadog* Claude3 CircieciPlatform Team800$ 5y-12250 / [ Jy-20632READY FOR DEV 1Prepare fallback with email for SSO for 'persistent name_id_formatoohesnetxWiwexceed Fontawesomepackage limitsMAINTENANCISReady for Dev©-20564 1 П ****=DescriptionEdit descriptionSubtasksAdd subtaskLinked work items0 SRD-6779 Unable to log in to SidekickIN PROGRESS MActivityWOrK lOAdd a commenttCan I get more info.?Fro bpc press M lo comment)Turn on microphone (% + d)9:54 AM Daily - PlatformActions s(ablDaily - Platform • now100% <78• Tue 14 Apr 9:54:28Chat8•С Ct: DepeActions sDependol• Far DepeD TasksInsights & Coachin.• DevD ux.IZ Improve Story~ DetailsNikolay NikolovReporterNikolay MikolovDevelopment #@ Open with VS Code|) Create branch12 commits|1 pull request5 days agcOPEN CouikkComponentsPlatformSuoerooueSidekickLabelsNoneStory point estimateD Al BookmarksGroup: Queriesteature fiag reieaseDeployed120065 15 .000=AI Review - Q1CTNIEUANAORCEVDeployed_190821o de0=No such column "Type' onentty 'Event"PLATTORM STARTLITYDeployed© -20384 05 П ****=Missed deletion of stale0 -40001aesese(Pipedrive) DI > status notlabeled, date data missingREOUCT CHURNNDeployed0 -20614 05 П ****=Stefka StoyanovaNikolay YankovSteliyan GeorgievNikolay Ivanov:#Lukas Kovalik9:04...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8578
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
Mute Stefka Stoyanova's microphone
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:54
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Turn on microphone (⌘ + d)
Your microphone is off.
meet.google.com / Daily - Platform
Platform Sprint 1 Q2 - Platform Tea@ Jy 19798 evaluation for ai activ+ New labHistoryBookmarksProfilesToolsWindow Helpgithub.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8~ Google Gemini...soccce Cdcsediie3.Run: Executes the evaluation viaasyncio.run(run_evaluation(...)),which orchestrates the collection,endpoint calling, and result scoringoeses.are there soem issueswith this PRShow thinking vYes, there are several issues and code smellsin this Pull Request, ranging from performancebottlenecks to architectural anti-patterns.Here is a breakdown of the most notableissues, categorized by impact:Performance & Async Anti-Patterns• Sequential Requests Defeating Async(Lines 86-106): Inside_collect_from_es, the code loopsthrough raw_samples and await sasession.post request for each sampleone by one. This means the script waitsfor one HTTP request to finish beforestarting the next, completely defeatingthe purpose of using aiohttp forasynchronous concurrency.• Fix: Collect these coroutines into alist and use asyncio.gather ()(ideally wrapped in anasyncio. Semaphore to avoidoverwhelming the endpoint) to run• Enter a prompt for GeminiPro vl1 OpenJy 19798 evaluation for ai activity types #468All commitsnikolaybiaivanov wants to merge 9 commits into master from JY-19798-evaluation-for-ai-activity-types [v src/eval/eval-ai-activity-type.py SQ Filter files...• E eval_output/ai_activity_type/eval…..> • ground_truth>I inoutE src/eval= enapolnts.pyE eval-ai-activity-type.pyE .gitignoreYour Jiminny chats aren't used to improve our models. Gemini is Alane can make mistakes, Inciualne aoout peodleYour privacy & GeminiSummarize page>.gitignore ?293299+329330< 40 lhl | Daily - Platform • now100% (' • Tue 14 Apr 9:54:28• 2 1 203 viewed•Awallind aporovali+330 UUUUU |Viewednarser = create narser AiActivitvTvoervaluator)parser.add_argument("--data-set",lyoe-sulndefault="initial,dest="data_set",helo="Dataset folder under eval outout/ai activity tvoe/eval datasets/ (default:initial)",parser.add_argument("--populate-ground-truth-Ulm",action="store_true",dest="populate_ground_truth_llm",parser.add_argument("_-ground-truth-llm-endpoint",nvoessur.default="OPENAI: gpt-5_2",dest="ground_truth_llm_endpoint",parser.add_argument("--compare-to-ground-truth",type=str,choices=["es", "1lm"],derault=es,dest="compare_to_ground_truth",aros = parser.parse arasAiActivityTypeEvaluator.DATA_SET = args.data_setA1Activitylypeevaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_(m)AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpointAiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truthasyncio.run(run_evaluation(AiActivityTypeEvaluator, args))+4 0000• Viewed••You made it to the end!...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8579
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
Mute Stefka Stoyanova's microphone
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:54
AM
Daily - Platform
Daily - Platform
Audio settings
Turn off microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is on.
meet.google.com / Daily - Platform
DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelpmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)E:1nbox (2)SRD-677C Jy 19798(7 Service-oir Font. AmrCostI Clhttps://jiminny.atlassian.net/jira/software/c/projects/JY/boards/372selectedissue=JY-20632Simir0 ProjectsB AWSE3 SSHE Datadog* Claude3 CircieciPlatform Team800$ 5y-12250 / [ Jy-20632READY FOR DEV 1Prepare fallback with email for SSO for 'persistent name_id_formatoohesnetxWiwexceed Fontawesomepackage limitsMAINTENANCISReady for Dev©-20564 1 П ****=DescriptionEdit descriptionSubtasksAdd subtaskLinked work items0 SRD-6779 Unable to log in to SidekickIN PROGRESS MActivityWOrK lOAod a comment.Can I get more info.?Status update...Fro bpc press M lo comment)9:54 AM Daily - PlatformActions s(ablDaily - Platform • now100% <478• Tue 14 Apr 9:54:336ChatС Ct: DepeActions sDependolD TasksInsights & Coachin.• DelD ux.IZ Improve Story~ DetailsNikolay NikolovAsuntoeReporter• Nikolay NikolovUerelo enecne@ Open with VS Code|) Create branch12 commits1 pull requestouikkComponentsPlatformSuoerooueSidekickLabelsNoneStory point estimate5 days agcOPEN C8•• Far DepeD Al BookmarksGroup: QueriesSUPPORT TICKETSDeployed0E -20505 15 + ****=AI Review - Q1CTNIEUANAORCEVDeployed_19082 1o de0=No such column "Type' onentty 'Event"PLATTORM STARTLITYDeployed© -20384 05 11 ****=Missed deletion of stale0 -40001a8ese(Pipedrive) DI > status notlabeled, date data missingREOUCT CHURNNDeployed0 -20614 05 П ****=:Stefka StoyanovaNikolay YankovSteliyan GeorgievNikolay Ivanov9:09...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8580
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
Mute Stefka Stoyanova's microphone
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:54
AM
Daily - Platform
Daily - Platform
Audio settings
Turn off microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is on.
meet.google.com / Daily - Platform
Platform Sprint 1 Q2 - Platform Tea@ Jy 19798 evaluation for ai activ+ New labHistoryBookmarksProfilesToolsWindow Helpgithub.com/jiminny/prophet/pull/468/changes#diff-d2f9fea9bc428b119ae26a66453e8eab7dd0782d4a4d93075ca5ce05903d77e8~ Google Gemini...soccce Cdcsediie3.Run: Executes the evaluation viaasyncio.run(run_evaluation(...)),which orchestrates the collection,endpoint calling, and result scoringoeses.are there soem issueswith this PRShow thinking vYes, there are several issues and code smellsin this Pull Request, ranging from performancebottlenecks to architectural anti-patterns.Here is a breakdown of the most notableissues, categorized by impact:Performance & Async Anti-Patterns• Sequential Requests Defeating Async(Lines 86-106): Inside_collect_from_es, the code loopsthrough raw_samples and await sasession.post request for each sampleone by one. This means the script waitsfor one HTTP request to finish beforestarting the next, completely defeatingthe purpose of using aiohttp forasynchronous concurrency.• Fix: Collect these coroutines into alist and use asyncio.gather ()(ideally wrapped in anasyncio. Semaphore to avoidoverwhelming the endpoint) to run• Enter a prompt for GeminiPro vl1 OpenJy 19798 evaluation for ai activity types #468All commitsnikolaybiaivanov wants to merge 9 commits into master from JY-19798-evaluation-for-ai-activity-types [v src/eval/eval-ai-activity-type.py SQ Filter files...• E eval_output/ai_activity_type/eval…..> • ground_truth>I inoutE src/eval= enapolnts.pyE eval-ai-activity-type.pyE .gitignoreYour Jiminny chats aren't used to improve our models. Gemini is Alane can make mistakes, Inciualne aoout peodleYour privacy & GeminiSummarize page>.gitignore ?293299+329330< 40 lhl | Daily - Platform • now100% [42' 8• Tue 14 Apr 9:54:33• 2 1 203 viewed•Awallind aporovall+330 UUUUU |Viewednarser = create narser AiActivitvTvoervaluator)parser.add_argument("--data-set",lyoe-sulndefault="initial,dest="data_set",helo="Dataset folder under eval outout/ai activity tvoe/eval datasets/ (default:initial)",parser.add_argument("--populate-ground-truth-Ulm",action="store_true",dest="populate_ground_truth_llm",parser.add_argument("_-ground-truth-llm-endpoint",nvoessur.default="OPENAI: gpt-5_2",dest="ground_truth_llm_endpoint",parser.add_argument("--compare-to-ground-truth",type=str,choices=["es", "1lm"],derault=es,dest="compare_to_ground_truth",aros = parser.parse arasAiActivityTypeEvaluator.DATA_SET = args.data_setA1Activitylypeevaluator.POPULATE_GROUND_TRUTH_LLM = bool(args.populate_ground_truth_(m)AiActivityTypeEvaluator.GROUND_TRUTH_LLM_ENDPOINT = args.ground_truth_llm_endpointAiActivityTypeEvaluator.COMPARE_TO_GROUND_TRUTH = args.compare_to_ground_truthasyncio.run(run_evaluation(AiActivityTypeEvaluator, args))+4 0000• Viewed••You made it to the end!...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8581
|
|
DiaFileEditViewTabsBookmarksHistoryExtensionsWindo DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelpmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)E:1nbox (2)SRD-677C7 (SRO-67C Jy 19798(7 Service-oir Font. Amr• COMTICHhttps://jiminny.atlassian.net/jira/software/c/projects/JY/boards/372selectedissue=JY-20632Simir0 ProjectsB AWSE3 SSHE Datadog* Claude3 CircieciPlatform Team800$ Jy-12250 / [ JY-20632READY FOR DEV 1oohesnetxWiwexceed Fontawesomepackage limitsMAINTENANCISReady for Dev@-205641 П ****:Prepare fallback with email for SSO for 'persistent name_id_formatDescriptionEdit descriptionSubtasksAdd subtask:Linked work items0 SRD-6779 Unable to log in to SidekickIN PROGRESS MActivityWOrK lOAdd a commenttCan I get more info.?Status update...Fro bpc press M lo comment)9:54 AM|Daily - PlatformActions s(ablDaily - Platform • now100% [8• Tue 14 Apr 9:54:35Chat8•С Ct: DepeActions sDependol• Far DepeD TasksInsights & Coachin.• DevD ux.#imorove Story~ DetailsNikolay NikolovAsuntoeReporter• Nikolay NikolovUerelo enecne@ Open with VS Code|) Create branch12 commits1 pull request1 build5 days agcUothComponentsPlatformSuoerooueSidekickLabelsNoneStory point estimateGroup: Queriesteature fiag reieaseDeployed0 -20ses 15 4 •***=AI Review - Q1CTNIHEUANAORCEVDeployed_19082 |1 0 •***=No such column "Type' onentty 'Event"PLATTORM STARTLTTYDeployed2038405 0000=Missed deletion of stalea8ese(Pipedrive) DI > status notD Al BookmarksStefka StoyanovaNikolay YankovSteliyan GeorgievNikolay Ivanovovalik:9:10...
|
NULL
|
NULL
|
NULL
|
8582
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
You can't unmute someone else
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
You can't unmute someone else
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:54
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform
DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelp(abl| Daily - Platform • now100% <478• Tue 14 Apr 9:54:38meet.google.com / Daily - PlatformNikolay Yankov (Presenting)Chat8•Inbox (2)PlactormSRD-677C Jy 19798(7 Service-oie Font. Anc• COMTICH• Actions :С Ct: DepeQ Actions :Dependolhttps://jiminny.atlassian.net/jira/software/c/projects/JY/boards/37• Sminn• ProjectiE SSHPlatform TeamQ Search board2800018READY FOR DEV 1IN DEV 4oeeretxWirexceed FontawesomeMASNTENANCSReady for Dev8-205641'*aa*Part? Automated reoorts)with Ask JiminnyAJREPORTSIn DevBE4-16400 115n****= 3Delays in CRM Sync|CUATFORM STABTUITYIn DevQJy-20553workfiow for automatedDependabot fixesEMPROVEMENT OF OUR EFFICHINCYIn Dev© 3-20551ru and swoiize wodieOS using new mobile use...MAINTENANCEIn DevOAД -37-20431El Dacadog * Claude3 CircieciA Sentryl0 TasksInsights & Coachin.• DevD ux.Label vQuick filtersComplete sprintCOOE REVIEW 3BLOCKEDPO ACCEPTANCEEvaluation for AI ActivityTypesAUTO-DETECTED ACTIVITY TYPECode Review9л-19798 15 + **•=AJ Reports > TrackingCode ReviewД J-20543 / 11 •*• =for SSO for 'persistent'name_id_formatREDUCE CHURNCode ReviewД 31-206320,5 ..• =• Far DepeD Al BookmarksGroup: QueriesDEPLOY &nx oownome during teleature fiag releaseBUPPORT TICKETSDeployed0 -20565 15 ****=AI Review - Q1Oeployed• -19982 1 @ •000=|no sucn column "lype onentity "Event"PLATFORM STARTUITYDeployed9 _20384 05 П •**•=|Missed deletion of staleREDUCE CHURN0 20619 1 11 ****=(Pipedrive) DI > status notStefka StoyanovaNikolay Yankovyan GeorgievNikolay lvanovLovalik9:54 AM Daily - Platform9:13...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8583
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Zoom in
Open in new window
Enter Full Screen
Stefka Stoyanova
Nikolay Yankov
Steliyan Georgiev
Nikolay Ivanov
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:54
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8584
|
|
Nikolay Yankov (Presenting)
Nikolay Yankov (Presen Nikolay Yankov (Presenting)
Nikolay Yankov (Presenting)
People
6
Take notes with Gemini
Take notes with Gemini
Gemini
Gemini
Unpin Nikolay Yankov's presentation from your main screen
You can't unmute someone else's presentation
More options for Nikolay Yankov
Zoom in
Open in new window
Enter Full Screen
Pin Stefka Stoyanova to your main screen
You can't unmute someone else
More options for Stefka Stoyanova
Stefka Stoyanova
Pin Nikolay Yankov to your main screen
Mute Nikolay Yankov's microphone
More options for Nikolay Yankov
Nikolay Yankov
Pin Steliyan Georgiev to your main screen
Mute Steliyan Georgiev's microphone
More options for Steliyan Georgiev
Steliyan Georgiev
Pin Nikolay Ivanov to your main screen
You can't unmute someone else
More options for Nikolay Ivanov
Nikolay Ivanov
You’re continuously framed
Backgrounds and effects
More options for Lukas Kovalik
Lukas Kovalik
Others might see more of your background. Click to view your full video.
9:54
AM
Daily - Platform
Daily - Platform
Audio settings
Turn on microphone
Video settings
Turn off camera
Nikolay Yankov is presenting
Send a reaction
Turn on captions
Raise hand (ctrl + ⌘ + h)
More options
Leave call
Meeting details
Chat with everyone
Meeting tools
Your microphone is off.
meet.google.com / Daily - Platform
DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelpmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)PlactormSRD-677X(7 (SRD-67:C Jy 19798(7 Service-oir Font. Amrhttps://jiminny.atlassian.net/jira/software/c/projects/JY/boards/377selectedissue=JY-18909• Siminm0 ProjectsE3 SSHPlatform TeamQ Search board200$ Jy-19240 / [ JY-18900READY FOR DEV 1SubtasksooeeneTxwirtexceed Fontawesomepackage limits(MAINTENANCISReady for Dev8-205641 П ****:%JY-20570(FE) Prepare HTML Template for PDFreport9JY-20571(AI) Create PDF from Panorama9 JY-20572(BE) Send emall for generated report(check design)9J1-20573(BE) Manage recipients for emallsending% JY-20574 [AI) Ensure PDF formatting is good% J7-20575work9JY-20576(FE) Add new generated report in theAl reports page(BE) Add flag in AI Reports list for9 JY-20577delete rights% JY-20578 [FE) Add delete button|9JY-20579(BE) Add new report type in filtersoptions% JY-20580 (FE) Rename column Shared% JY-20583 (BE) Add period to cron job9Jy-20584(BE) Change search report in cron jobEl DatadogPriority= Medium= Medium= Medium= Medium= Medium= Medium= Medium= Medium= Medium= Medium= Medium= Medium= Medium= Medium= MediumCostI ClActionsNikolay Y…Steliyan .@ Lukas Ko...* Lukas Ko...Steliyan…..3 Stellyan ...Nikolay Y..Lukas Ko...Nikolay Y..Nikolay Y…Steliyan...Lukas Ko...LUxaS KO..••-+77% DoneDONE MDONEIN DEVDONE MDONE VIDONEDONEIN DEV VDONE MDONE y9:54 AM Daily - PlatformDaily - Platform • now100% <478• Tue 14 Apr 9:54:47Chat8•CE: DeperDependol• Far DepeD TasksInsights & Coachin• DelD ux.© 2Xz Improve Story~ DetailsAssignee 332 Stellyan GeorgievAsentoeReporter2 Galya DimitrovaUeyele encne@ Open with VS Code|3 branches32 commits1 pull request1 bulld fallingRelessesProduction2 See all deploymentsComponentsPlatformSub-ProductAoorcons5 days agcUrEhD Al BookmarksGroup: QueriesDEPLOY &leature fiag releaseDeployed0 -20585 15 + ****=AI Review - Q1CTNMEUANANORCORDeployed_19982 |1 0 •*=•=No such column "Type' onentty 'Event"PLATTORM STABTLITYDeployed9 -20384 05 ^ ****=|Missed deletion of stale# *ee(Pipedrive] DI > status notlabeled, date data missingREDUCE CHURNDeployed0 -20614 05 П ****=:Stefka StoyanovaNikolay Yankovan GeorgievNikolay Ivanovvalik• 9:22...
|
Dia
|
Work: Meet - Daily - Pla…
|
NULL
|
8585
|
|
JY-20543 add AJ reports User pilot tracking by Lak JY-20543 add AJ reports User pilot tracking by LakyLak · Pull Request #11932 · jiminny/app
Platform Sprint 1 Q2 - Platform Team - Scrum Board - Jira
Platform Sprint 1 Q2 - Platform Team - Scrum Board - Jira
Jy 19798 evaluation for ai activity types by nikolaybiaivanov · Pull Request #468 · jiminny/prophet
Jy 19798 evaluation for ai activity types by nikolaybiaivanov · Pull Request #468 · jiminny/prophet
New Tab
New Tab
Close tab
New Tab
Customize sidebar
Close Google Gemini (⌃X)
Tabs from other devices
Open history (⇧⌘H)
Open bookmarks (⌘B)
AI Chat settings
Close
WORK, Google Account: [EMAIL]
Main menu
New Chat
Open menu for conversation actions.
DiaFileEditViewTabsBookmarksHistoryExtensionsWindowHelpmeet.google.com / Daily - PlatformNikolay Yankov (Presenting)PlactormSRD-677X(7 (SRD-67:C Jy 19798(7 Service-oir Font. Amrhttps://jiminny.atlassian.net/jira/software/c/projects/JY/boards/377selectedissue=JY-18909Sminm0 ProjectsE3 SSHPlatform TeamQ Search board200$ Jy-19240 / [ JY-18900READY FOR DEV 1SubtasksooeeneTxwirtexceed Fontawesomepackage limits(MAINTENANCISReady for Dev8-205641 П ****:%JY-20570(FE) Prepare HTML Template for PDFreport9JY-20571(AI) Create PDF from Panorama9 JY-20572(BE) Send emall for generated report(check design)9J1-20573(BE) Manage recipients for emallsending% JY-20574 [AI) Ensure PDF formatting is good% J7-20575work9JY-20576(FE) Add new generated report in theAl reports page(BE) Add flag in AI Reports list for9 JY-20577delete rights% JY-20578 [FE) Add delete button|9JY-20579(BE) Add new report type in filtersoptions% JY-20580 (FE) Rename column SharedEl DatadogPriority= Medium= Medium= Medium= Medium= Medium= Medium= Medium= Medium= Medium= Medium= Medium= Medium= Medium% JY-20583 (BE) Add period to cron job= Medium9Jy-20584(BE) Change search report in cron job= MeciumCostI ClActionsNikolay Y…Steliyan .@ Lukas Ko...* Lukas Ko...Steliyan…..3 Stellyan ...Lukas Ko...Nikolay Y..Nikolay Y…Steliyan...Lukas Ko...LUxaS KO..••-+77% DoneDONE MDONEIN DEVDONE MDONE VIDONEDONEIN DEV VDONE MDONE y9:54 AM Daily - PlatformDaily - Platform • now100% [8• Tue 14 Apr 9:54:52Chat8•CE: DepeDependol• Far DepeD TasksInsights & Coachin• DelD ux.© 2Xz Improve Story~ DetailsAssignee 33 Stellyan GeorgievAsentoeReporter2 Galya DimitrovaUeyele encne@ Open with VS Code|3 branches32 commits1 pull request1 bulld fallingRelessesProduction2 See all deploymentsComponentsPlatformSub-ProductAoorcons5 days agcUrEhD Al BookmarksGroup: QueriesDEPLOY &leature fiag releaseDeployed12006515 * *00=AI Review - Q1CTNMEUANANORCORDeployed-19982 1 @ ...•=No such column "Type' onentty 'Event"PLATTORM STABTUTTYDeployed9 -20384 05 11 ****=Missed deletion of staleaesese(Pipedrive) DI › status notlabeled, date data missingREDUCE CHURNDeployed0 -20614 05 П ****=:Stefka StoyanovaNikolay YankovSteliyan GeorgievNikolay Ivanovas Kovalik9:28...
|
Firefox
|
Work — Mozilla Firefox
|
NULL
|
8586
|