Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update bot to match the internal repo #5

Merged
merged 4 commits into from
Oct 7, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added .DS_Store
Binary file not shown.
41 changes: 41 additions & 0 deletions .github/workflows/daily_run_simple_bot.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Daily Build Simple Bot

on:
schedule:
- cron: "0 15 * * *" # every day, 11:00 EST

# Daily job to run the simple forecast bot
jobs:
daily_build:
runs-on: ubuntu-latest #determines the machine that will run the job - keep as is
steps: # sets up the steps that will be run in order
# setup repository with all necessary dependencies - keep as is
- name: Check out repository
uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install poetry
uses: snok/install-poetry@v1
with:
virtualenvs-create: true
virtualenvs-in-project: true
installer-parallel: true
- name: Load cached venv
id: cached-poetry-dependencies
uses: actions/cache@v2
with:
path: .venv
key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}
- name: Install dependencies
run: poetry install --no-interaction --no-root # note that this will install dependencies for the main bot, which is a bit more than what's needed for the simple bot. But that shouldn't matter much.
# run your bot
- name: Run bot
run: |
poetry run python simple-forecast-bot.py
# this reads the environment variables from the github repository.
# Store under Settings --> Secrets and variables --> Actions
env:
METACULUS_TOKEN: ${{ secrets.METACULUS_TOKEN }} # replace this with the name of the variable under which you stored your own Metaculus token
PERPLEXITY_API_KEY: ${{ secrets.PERPLEXITY_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
name: run daily
name: PR Check

on:
schedule:
- cron: "0 0 * * *" # every day
workflow_dispatch:
pull_request:
branches: [main]

jobs:
build:
pr_check:
runs-on: ubuntu-latest
steps:
- name: Check out repository
Expand All @@ -27,14 +26,11 @@ jobs:
path: .venv
key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}
- name: Install dependencies
if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
run: poetry install --no-interaction --no-root
- name: Run the script
if: always()
run: echo "poetry run python forecast_bot.py"
env: # Or as an environment variable
- name: Run simple bot
run: |
poetry run python simple-forecast-bot.py
env:
METACULUS_TOKEN: ${{ secrets.METACULUS_TOKEN }}
PERPLEXITY_API_KEY: ${{ secrets.PERPLEXITY_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
TOURNAMENT_ID: ${{ vars.TOURNAMENT_ID }}
API_BASE_URL: ${{ vars.API_BASE_URL }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
63 changes: 55 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,59 @@
# Simple forecasting bot
# Metaculus forecasting bot

This is a very simple forecasting bot that uses an LLM to forecast on a Metaculus tournament. It lists all questions from the tournament, uses Perplexity.ai to search for up-to-date news about the questions, and then uses ChatGPT to make forecasts.
This repository contains the bots Metaculus uses as a baseline for the [AI Forecasting Tournament](https://www.metaculus.com/aib/). In addition, it contains a simple bot meant to get you started with creating your own bot.

You need to set these secrets and environment variables:

- `METACULUS_TOKEN` - register your bot to get a token [here](https://www.metaculus.com/aib/)
- `TOURNAMENT_ID` - the ID of the tournament your bot should forecast on
- `OPENAI_API_KEY` - used to access ChatGPT and make forecasts on the questions
- `PERPLEXITY_API_KEY` - used to search for up-to-date information about the questions
## Getting started

It uses GitHub Actions to schedule it for running daily (you can see more about that in [daily_run.yaml](.github/workflows/daily_run.yaml))
### Setting up the repository
Fork this repository (click the "Fork" button in the top right corner). You don't have to fork this repository. But if you do, you can make use of the github actions to run your own bot automatically.

Clone the repository. Find your terminal and run the following commands:
```bash
git clone https://github.com/Metaculus/metac-bot
```

If you forked the repository first, you have to replace the url in the `git clone` command with the url to your fork. Just go to your forked repository and copy the url from the address bar in the browser.

### Installing dependencies
Make sure you have python and [poetry](https://python-poetry.org/docs/#installing-with-pipx) installed (poetry is a python package manager).

Inside the terminal, go to the directory you cloned the repository into and run the following command:
```bash
poetry install
```
to install all required dependencies.

### Setting environment variables

#### Locally
Running the bot requires various environment variables. If you run the bot locally, the easiest way to set them is to create a file called `.env` in the root directory of the repository and add the variables in the following format:
```bash
METACULUS_TOKEN=1234567890 # register your bot to get a here: https://www.metaculus.com/aib/
OPENAI_API_KEY=1234567890
PERPLEXITY_API_KEY=1234567890 # optional, if you want to use perplexity.ai
ANTHROPIC_API_KEY=1234567890
```
#### Github Actions
If you want to automate running the bot using github actions, you have to set the environment variables in the github repository settings.
Go to (Settings -> Secrets and variables -> Actions). Set API keys as secrets and the tournament id and API base URL as variables.

For this simple bot, `TOURNAMENT_ID` and `API_BASE_URL` are simply hard-coded in the script and can be changed in the code itself.

## Running the bot

To run the simple bot, execute the following command in your terminal:
```bash
poetry run python simple-forecast-bot.py
```
Make sure to set the environment variables as described above and to set the parameters in the code to your liking. In particular, to submit predictions, make sure that `submit_predictions` is set to `True`.

## Automating the bot using Github Actions

Github can automatically run code in a repository. To that end, you need to fork this repository. You also need to set the secrets and environment variables in the github repository settings as explained above.

Automation is handled in the `.github/workflows/` folder.

The `pr_check.yaml` file is responsible for triggering a test run every time a pull request is made to the main branch. This is useful for development and testing.

The `daily_run_simple_bot.yaml` file runs the simple bot every day (note that since `submit_predictions` is set to `False` in the script by default, no predictions will actually be posted). The `daily_run_simple_bot.yaml` file contains various comments and explanations. You should be able to simply copy this file and modify it to run your own bot.
Loading
Loading