Internal Server Error (500) when using Gemini API with Inspect framework

[lennijusten]

Description of the bug:

Description

I'm consistently encountering an Internal Server Error (HTTP 500) when trying to use the Google Gemini API through the Inspect evaluation framework. This error occurs during the generate_content call. I'm currently on the free trial.

Steps to Reproduce

1. Set up an evaluation using the Inspect framework
2. Configure the evaluation to use the Gemini model (in my case, google/gemini-1.5-pro)
3. Run the evaluation

Error Message

InternalServerError: 500 An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting

Environment

• Operating System: MacOS Sonoma 14.5
• Python version: 3.12.4

Package Versions

• google-ai-generativelanguage: 0.6.6
• google-api-core: 2.19.2
• google-api-python-client: 2.143.0
• google-auth: 2.34.0
• google-auth-httplib2: 0.2.0
• google-generativeai: 0.7.2
• googleapis-common-protos: 1.65.0

Full error traceback:

╭─ benchmarks/gpqa (78 samples): google/gemini-1.5-pro ─────────────────────────────────────────────────────────────────────╮
│ ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮   dataset: (samples) │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │       scorer: choice │
│ │ in task_run                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in task_run_sample                                                                               │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in __call__                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in task_run_sample                                                                               │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in solve                                                                                         │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in generate                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in task_generate                                                                                 │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in generate                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in _generate                                                                                     │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/as… │                      │
│ │ in async_wrapped                                                                                 │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/as… │                      │
│ │ in __call__                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/as… │                      │
│ │ in iter                                                                                          │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/_u… │                      │
│ │ in inner                                                                                         │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/__… │                      │
│ │ in <lambda>                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.1… │                      │
│ │ in result                                                                                        │                      │
│ │                                                                                                  │                      │
│ │   446 │   │   │   │   if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:                     │                      │
│ │   447 │   │   │   │   │   raise CancelledError()                                                 │                      │
│ │   448 │   │   │   │   elif self._state == FINISHED:                                              │                      │
│ │ ❱ 449 │   │   │   │   │   return self.__get_result()                                             │                      │
│ │   450 │   │   │   │                                                                              │                      │
│ │   451 │   │   │   │   self._condition.wait(timeout)                                              │                      │
│ │   452                                                                                            │                      │
│ │                                                                                                  │                      │
│ │ /opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.1… │                      │
│ │ in __get_result                                                                                  │                      │
│ │                                                                                                  │                      │
│ │   398 │   def __get_result(self):                                                                │                      │
│ │   399 │   │   if self._exception:                                                                │                      │
│ │   400 │   │   │   try:                                                                           │                      │
│ │ ❱ 401 │   │   │   │   raise self._exception                                                      │                      │
│ │   402 │   │   │   finally:                                                                       │                      │
│ │   403 │   │   │   │   # Break a reference cycle with the exception in self._exception            │                      │
│ │   404 │   │   │   │   self = None                                                                │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/as… │                      │
│ │ in __call__                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in generate                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in generate                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/gene… │                      │
│ │ in generate_content_async                                                                        │                      │
│ │                                                                                                  │                      │
│ │   382 │   │   │   │   │   )                                                                      │                      │
│ │   383 │   │   │   │   return await generation_types.AsyncGenerateContentResponse.from_aiterato   │                      │
│ │   384 │   │   │   else:                                                                          │                      │
│ │ ❱ 385 │   │   │   │   response = await self._async_client.generate_content(                      │                      │
│ │   386 │   │   │   │   │   request,                                                               │                      │
│ │   387 │   │   │   │   │   **request_options,                                                     │                      │
│ │   388 │   │   │   │   )                                                                          │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/ai/g… │                      │
│ │ in generate_content                                                                              │                      │
│ │                                                                                                  │                      │
│ │    403 │   │   self._client._validate_universe_domain()                                          │                      │
│ │    404 │   │                                                                                     │                      │
│ │    405 │   │   # Send the request.                                                               │                      │
│ │ ❱  406 │   │   response = await rpc(                                                             │                      │
│ │    407 │   │   │   request,                                                                      │                      │
│ │    408 │   │   │   retry=retry,                                                                  │                      │
│ │    409 │   │   │   timeout=timeout,                                                              │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/api_… │                      │
│ │ in retry_wrapped_func                                                                            │                      │
│ │                                                                                                  │                      │
│ │   227 │   │   │   sleep_generator = exponential_sleep_generator(                                 │                      │
│ │   228 │   │   │   │   self._initial, self._maximum, multiplier=self._multiplier                  │                      │
│ │   229 │   │   │   )                                                                              │                      │
│ │ ❱ 230 │   │   │   return await retry_target(                                                     │                      │
│ │   231 │   │   │   │   functools.partial(func, *args, **kwargs),                                  │                      │
│ │   232 │   │   │   │   predicate=self._predicate,                                                 │                      │
│ │   233 │   │   │   │   sleep_generator=sleep_generator,                                           │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/api_… │                      │
│ │ in retry_target                                                                                  │                      │
│ │                                                                                                  │                      │
│ │   157 │   │   # This function explicitly must deal with broad exceptions.                        │                      │
│ │   158 │   │   except Exception as exc:                                                           │                      │
│ │   159 │   │   │   # defer to shared logic for handling errors                                    │                      │
│ │ ❱ 160 │   │   │   _retry_error_helper(                                                           │                      │
│ │   161 │   │   │   │   exc,                                                                       │                      │
│ │   162 │   │   │   │   deadline,                                                                  │                      │
│ │   163 │   │   │   │   sleep,                                                                     │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/api_… │                      │
│ │ in _retry_error_helper                                                                           │                      │
│ │                                                                                                  │                      │
│ │   209 │   │   │   RetryFailureReason.NON_RETRYABLE_ERROR,                                        │                      │
│ │   210 │   │   │   original_timeout,                                                              │                      │
│ │   211 │   │   )                                                                                  │                      │
│ │ ❱ 212 │   │   raise final_exc from source_exc                                                    │                      │
│ │   213 │   if on_error_fn is not None:                                                            │                      │
│ │   214 │   │   on_error_fn(exc)                                                                   │                      │
│ │   215 │   if deadline is not None and time.monotonic() + next_sleep > deadline:                  │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/api_… │                      │
│ │ in retry_target                                                                                  │                      │
│ │                                                                                                  │                      │
│ │   152 │                                                                                          │                      │
│ │   153 │   for sleep in sleep_generator:                                                          │                      │
│ │   154 │   │   try:                                                                               │                      │
│ │ ❱ 155 │   │   │   return await target()                                                          │                      │
│ │   156 │   │   # pylint: disable=broad-except                                                     │                      │
│ │   157 │   │   # This function explicitly must deal with broad exceptions.                        │                      │
│ │   158 │   │   except Exception as exc:                                                           │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/api_… │                      │
│ │ in __await__                                                                                     │                      │
│ │                                                                                                  │                      │
│ │    85 │   │   │   response = yield from self._call.__await__()                                   │                      │
│ │    86 │   │   │   return response                                                                │                      │
│ │    87 │   │   except grpc.RpcError as rpc_error:                                                 │                      │
│ │ ❱  88 │   │   │   raise exceptions.from_grpc_error(rpc_error) from rpc_error                     │                      │
│ │    89                                                                                            │                      │
│ │    90                                                                                            │                      │
│ │    91 class _WrappedStreamResponseMixin(Generic[P], _WrappedCall):                               │                      │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯                      │
│ InternalServerError: 500 An internal error has occurred. Please retry or report in                                        │
│ https://developers.generativeai.google/guide/troubleshooting  

Actual vs expected behavior:

No response

Any other information you'd like to share?

No response

[gmKeshari]

Hi @lennijusten ,

Have you been facing this issue since the beginning ? I mean is it like you were getting the response earlier and now facing this issue?

Try upgrading the google.generative library once and then run your commands :

pip install -q --upgrade google-generativeai

Please let us know if you are still having the issue.

[lennijusten]

Yes. It's been a semi-persistent problem throughout my usage of Gemini. Sometimes it works, but then after some number of requests or tokens, I get the Internal Server Error (500). The issue is also discussed here and here on Reddit.

I ran pip install -q --upgrade google-generativeai which upgraded me from google-generativeai==0.6.6 to google-generativeai==0.8.2 but the issue persists.

If there is some kind of rate limit happening it would be useful for the error code to reflect that.

With the current workings, I'm just burning tokens for nothing since my task never completes before the error.

[gmKeshari]

Hi @lennijusten . Sometimes we see Internal Server Error (500) because there are too many requests coming, which results in an unexpected error on Google's side. You can refer to this Troubleshooting doc "https://ai.google.dev/gemini-api/docs/troubleshooting?lang=python". I recommend temporarily switching to another model (e.g. from Gemini 1.5 Pro to Gemini 1.5 Flash) and see if it works.

Internal Server Error (500) is not related to rate limit, but free tier does have rate limits based model variation. Please refer to this doc : "https://ai.google.dev/gemini-api/docs/models/gemini" or "https://ai.google.dev/pricing"

If you want to upgrade to "Pay-as-you-go" tier, you need to set up a billing account.

[MarkDaoust]

Hi, thanks for the report.

There have been cases where bad inputs cause 500s. But it sounds like it wasn't consistent about which requests cause the error.

Like @gmKeshari said the service used to throw 500 errrors when it's overloaded (I think they've changed that to a clearer error)

For intermittent errors, the SDK supports a 'retry' argument:

import google.generativeai as genai
from google.api_core import retry

model = genai.GenerativeModel('gemini-2.0-flash')

# For convenience, a simple wrapper to let the SDK handle error retries
def generate_with_retry(model, prompt):
  return model.generate_content(prompt, request_options={'retry':retry.Retry()})

I think the default limit is 5min, but it's configurable.

[pcalatayud-prog]

I have a billing account, but I'm facing the same error. It suddenly appeared... You know what I mean? It was working a few hours ago, but now it isn't. I'm making many async I/O calls, and it was functioning properly before.

[Timiolowo]

I am currently facing the issue, any solution yet?

[kiankyars]

me too

[reachgvm]

Yes, it is happening for me too.