Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate test script by LLM #20

Merged
merged 2 commits into from
Sep 19, 2024
Merged

Generate test script by LLM #20

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b37625e
Generate test script by LLM
Yuankai619 Sep 18, 2024
7579b24
Set testMatch and cancel some projects
Yuankai619 Sep 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 14 additions & 10 deletions test1/playwright.config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,17 @@ import { defineConfig, devices } from '@playwright/test';
*/
export default defineConfig({
testDir: './tests',
timeout: 6000,
testMatch:[
'basic.spec.ts',
'llm4.spec.ts'
],
timeout: 8000,
/* Run tests in files in parallel */
fullyParallel: true,
/* Fail the build on CI if you accidentally left test.only in the source code. */
forbidOnly: !!process.env.CI,
/* Retry on CI only */
retries: process.env.CI ? 2 : 0,
retries: process.env.CI ? 2 :2,
/* Opt out of parallel tests on CI. */
workers: process.env.CI ? 1 : undefined,
/* Reporter to use. See https://playwright.dev/docs/test-reporters */
Expand All @@ -39,15 +43,15 @@ export default defineConfig({
use: { ...devices['Desktop Chrome'] },
},

{
name: 'firefox',
use: { ...devices['Desktop Firefox'] },
},
// {
// name: 'firefox',
// use: { ...devices['Desktop Firefox'] },
// },

{
name: 'webkit',
use: { ...devices['Desktop Safari'] },
},
// {
// name: 'webkit',
// use: { ...devices['Desktop Safari'] },
// },

/* Test against mobile viewports. */
// {
Expand Down
45 changes: 44 additions & 1 deletion test1/prompts_record.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,4 +92,47 @@ bug: 一開始談出的選擇視窗樣式錯誤，看不到任何的文字和符
<script src="scripts.js"></script>
</body>
</html>
```
```

---
# 以下是測試prompt

## Prompt1
現在要用playwright進行端對端測試，針對每一個需求進行最詳盡的測試並且給我，預設使用typerscript 去生成playwright腳本，對於每個測試腳本要註解是測試的需求是哪個，並且盡量用data-testid和aria-label去做selector，給我完整的測試腳本

## Prompt2
生成的測試有幾個錯誤
每個動作是隨機的，所以沒辦法透過按照順序按格子模擬玩遊戲，因為有可能已經被電腦選走了
還有每個動作電腦回應都會停頓600ms，所以玩遊戲的過程中都要先等電腦回應後再動作
還有測試上沒有測試到遊戲結束後判斷輸贏是不是合法的

## Prompt3
在測試should randomly decide who goes first有錯誤，因為先手是隨機決定的，因此有可能是後手也有可能是先手，但是判斷卻是空格要小於9，如果是先手的話就會出錯
還有測試should display "Congratulations! You win" when player wins有錯誤，因為電腦是隨機選格子，所以如果直接模擬寫死的步驟碰巧被電腦選走就會出錯
should display "You lose! Try again" when player loses也是一樣，沒辦法透過簡單寫死的步驟讓電腦贏
should display "It\'s a tie!" when there is a tie也是，寫死的步驟會難以保證結果是平手的
should verify if the game ends with a correct legal win/loss condition也是沒辦法透過寫死的步驟保證這個是贏局或輸局，也有可能平手或是還沒結束
should restart game after confirming result 也是一樣的問題，寫死的步驟不能確保遊戲結束

## Prompt4
在should randomly decide who goes first會有錯誤: Error: expect(received).toBeGreaterThanOrEqual(expected)

Matcher error: received value must be a number or bigint

Received has type: object
Received has value: {Symbol(async_id_symbol): 5177, Symbol(trigger_async_id_symbol): 5176, Symbol(kResourceStore): undefined}
還有should not allow painting over existing moves會有錯誤:
Error: expect(received).toBeLessThan(expected)

Matcher error: received value must be a number or bigint

Received has type: object
Received has value: {Symbol(async_id_symbol): 7091, Symbol(trigger_async_id_symbol): 7090, Symbol(kResourceStore): undefined}
還有should restart game after confirming result有錯誤:
Error: expect(received).toBe(expected) // Object.is equality

Expected: 9
Received: {Symbol(async_id_symbol): 20900, Symbol(trigger_async_id_symbol): 20899, Symbol(kResourceStore): undefined}
還有在測試最後結果要輸、贏、平手的測試遊玩過程雖然正確但是電腦是隨機的，所以真的輸贏情形是無法預測的，所以應該要改成由玩遊戲然後判斷遊戲結果和board上面的紀錄是不是符合輸贏的條件才對
## Prompt5
還是有一些錯，在測試should display "It\'s a tie!" when there is a tie、should display "You lose! Try again" when player loses、其實是在測試同個動作，所以可以結合成一個就好
146 changes: 146 additions & 0 deletions test1/tests/llm0.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
import { test, expect } from '@playwright/test';

test.describe('Tic-Tac-Toe Game', () => {

// 測試需求 1: 確認遊戲開始時會顯示選擇符號的彈跳視窗
test('should display symbol choice modal at the beginning', async ({ page }) => {
// Navigate to the game
await page.goto('http://localhost:8080/');

// 確認彈出視窗顯示，且標題正確
const modalTitle = await page.locator('[data-testid="modal-title"]');
await expect(modalTitle).toHaveText('選擇你的符號');

// 確認按鈕 X 和 O 存在
await expect(page.locator('[data-testid="choose-X"]')).toBeVisible();
await expect(page.locator('[data-testid="choose-O"]')).toBeVisible();
});

// 測試需求 2: 玩家選擇 X 或 O 並開始遊戲
test('should start game after symbol selection', async ({ page }) => {
await page.goto('http://localhost:8080/');

// 點擊選擇 X 按鈕
await page.click('[data-testid="choose-X"]');

// 確認棋盤可見，並且沒有選擇彈出視窗
await expect(page.locator('[data-testid="game-board"]')).toBeVisible();
await expect(page.locator('[data-testid="symbol-choice-modal"]')).toBeHidden();
});

// 測試需求 3: 隨機決定玩家或電腦誰先畫格子
test('should randomly decide who goes first', async ({ page }) => {
await page.goto('http://localhost:8080/');

// 選擇 X 開始遊戲
await page.click('[data-testid="choose-X"]');

// 確認遊戲啟動後有至少一個棋格被選中（電腦可能會先行動）
const emptyCells = await page.locator('.cell:empty');
expect(emptyCells.count()).toBeLessThan(9); // 確認有格子已被畫上
});

// 測試需求 4: 玩家與電腦交互畫格子，不允許覆蓋已畫的格子
test('should not allow painting over existing moves', async ({ page }) => {
await page.goto('http://localhost:8080/');

// 玩家選擇 X 開始遊戲
await page.click('[data-testid="choose-X"]');

// 玩家選擇格子 0
const cell0 = page.locator('[data-testid="cell-0"]');
await cell0.click();

// 試圖再次點擊同一格子
await cell0.click();

// 確認內容未改變
await expect(cell0).toHaveText('X');
});

// 測試需求 5: 玩家獲勝時彈出視窗顯示 "恭喜!你贏了"
test('should display "Congratulations! You win" when player wins', async ({ page }) => {
await page.goto('http://localhost:8080/');

// 選擇 X 開始遊戲
await page.click('[data-testid="choose-X"]');

// 模擬玩家贏局，依次選擇 0, 1, 2
await page.click('[data-testid="cell-0"]'); // Player
await page.click('[data-testid="cell-3"]'); // Computer (random)
await page.click('[data-testid="cell-1"]'); // Player
await page.click('[data-testid="cell-4"]'); // Computer (random)
await page.click('[data-testid="cell-2"]'); // Player wins

// 確認贏的訊息彈出
await expect(page.locator('[data-testid="result-message"]')).toHaveText('恭喜!你贏了');
});

// 測試需求 6: 玩家失敗時彈出視窗顯示 "你輸了!下次再努力"
test('should display "You lose! Try again" when player loses', async ({ page }) => {
await page.goto('http://localhost:8080/');

// 選擇 X 開始遊戲
await page.click('[data-testid="choose-X"]');

// 模擬電腦贏局，依次選擇使電腦獲勝的步驟
await page.click('[data-testid="cell-0"]'); // Player
await page.click('[data-testid="cell-3"]'); // Computer
await page.click('[data-testid="cell-1"]'); // Player
await page.click('[data-testid="cell-4"]'); // Computer
await page.click('[data-testid="cell-8"]'); // Player
await page.click('[data-testid="cell-5"]'); // Computer wins

// 確認輸的訊息彈出
await expect(page.locator('[data-testid="result-message"]')).toHaveText('你輸了!下次再努力');
});

// 測試需求 7: 平局時顯示 "這局平手!"
test('should display "It\'s a tie!" when there is a tie', async ({ page }) => {
await page.goto('http://localhost:8080/');

// 選擇 X 開始遊戲
await page.click('[data-testid="choose-X"]');

// 模擬平局
await page.click('[data-testid="cell-0"]'); // Player
await page.click('[data-testid="cell-1"]'); // Computer
await page.click('[data-testid="cell-2"]'); // Player
await page.click('[data-testid="cell-4"]'); // Computer
await page.click('[data-testid="cell-3"]'); // Player
await page.click('[data-testid="cell-5"]'); // Computer
await page.click('[data-testid="cell-7"]'); // Player
await page.click('[data-testid="cell-6"]'); // Computer
await page.click('[data-testid="cell-8"]'); // Player

// 確認平局訊息彈出
await expect(page.locator('[data-testid="result-message"]')).toHaveText('這局平手!');
});

// 測試需求 8: 玩家確認結果後遊戲重啟
test('should restart game after confirming result', async ({ page }) => {
await page.goto('http://localhost:8080/');

// 選擇 X 開始遊戲
await page.click('[data-testid="choose-X"]');

// 模擬玩家贏局
await page.click('[data-testid="cell-0"]'); // Player
await page.click('[data-testid="cell-3"]'); // Computer (random)
await page.click('[data-testid="cell-1"]'); // Player
await page.click('[data-testid="cell-4"]'); // Computer (random)
await page.click('[data-testid="cell-2"]'); // Player wins

// 確認贏的訊息彈出
await expect(page.locator('[data-testid="result-message"]')).toHaveText('恭喜!你贏了');

// 點擊確認按鈕重新開始遊戲
await page.click('[data-testid="confirm-result"]');

// 確認遊戲已重新開始
await expect(page.locator('[data-testid="game-board"]')).toBeVisible();
const emptyCells = await page.locator('.cell:empty');
expect(emptyCells.count()).toBe(9); // 所有格子應該都是空的
});

});
134 changes: 134 additions & 0 deletions test1/tests/llm1.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
import { test, expect } from '@playwright/test';

// Helper function to simulate player moves
async function makeMove(page, cellIndex) {
const cell = page.locator(`[data-testid="cell-${cellIndex}"]`);
const isCellEmpty = await cell.evaluate(node => node.textContent === '');

if (isCellEmpty) {
await cell.click();
await page.waitForTimeout(600); // 等待電腦做出回應
}
}

test.describe('Tic-Tac-Toe Game', () => {

// 測試需求 1: 確認遊戲開始時會顯示選擇符號的彈跳視窗
test('should display symbol choice modal at the beginning', async ({ page }) => {
await page.goto('http://localhost:8080/');

const modalTitle = await page.locator('[data-testid="modal-title"]');
await expect(modalTitle).toHaveText('選擇你的符號');
await expect(page.locator('[data-testid="choose-X"]')).toBeVisible();
await expect(page.locator('[data-testid="choose-O"]')).toBeVisible();
});

// 測試需求 2: 玩家選擇 X 或 O 並開始遊戲
test('should start game after symbol selection', async ({ page }) => {
await page.goto('http://localhost:8080/');
await page.click('[data-testid="choose-X"]');

await expect(page.locator('[data-testid="game-board"]')).toBeVisible();
await expect(page.locator('[data-testid="symbol-choice-modal"]')).toBeHidden();
});

// 測試需求 3: 隨機決定玩家或電腦誰先畫格子
test('should randomly decide who goes first', async ({ page }) => {
await page.goto('http://localhost:8080/');
await page.click('[data-testid="choose-X"]');

await page.waitForTimeout(600); // 等待電腦可能的回應
const emptyCells = await page.locator('.cell:empty');
expect(emptyCells.count()).toBeLessThan(9); // 確認有格子已被畫上
});

// 測試需求 4: 玩家與電腦交互畫格子，不允許覆蓋已畫的格子
test('should not allow painting over existing moves', async ({ page }) => {
await page.goto('http://localhost:8080/');
await page.click('[data-testid="choose-X"]');

await makeMove(page, 0); // 玩家選擇格子 0

const cell0 = page.locator('[data-testid="cell-0"]');
await cell0.click(); // 試圖再次點擊已選的格子
await expect(cell0).toHaveText('X');
});

// 測試需求 5: 玩家獲勝時顯示 "恭喜!你贏了"
test('should display "Congratulations! You win" when player wins', async ({ page }) => {
await page.goto('http://localhost:8080/');
await page.click('[data-testid="choose-X"]');

await makeMove(page, 0); // 玩家選擇格子 0
await makeMove(page, 4); // 玩家選擇格子 4
await makeMove(page, 8); // 玩家選擇格子 8

// 檢查是否出現勝利訊息
await expect(page.locator('[data-testid="result-message"]')).toHaveText('恭喜!你贏了');
});

// 測試需求 6: 玩家失敗時顯示 "你輸了!下次再努力"
test('should display "You lose! Try again" when player loses', async ({ page }) => {
await page.goto('http://localhost:8080/');
await page.click('[data-testid="choose-X"]');

// 玩家和電腦隨機下棋，這裡模擬輸局
await makeMove(page, 0); // 玩家選擇格子 0
await makeMove(page, 2); // 玩家選擇格子 2
await makeMove(page, 8); // 玩家選擇格子 8

// 確認失敗訊息
await expect(page.locator('[data-testid="result-message"]')).toHaveText('你輸了!下次再努力');
});

// 測試需求 7: 平局時顯示 "這局平手!"
test('should display "It\'s a tie!" when there is a tie', async ({ page }) => {
await page.goto('http://localhost:8080/');
await page.click('[data-testid="choose-X"]');

await makeMove(page, 0); // 玩家選擇格子 0
await makeMove(page, 1); // 玩家選擇格子 1
await makeMove(page, 2); // 玩家選擇格子 2
await makeMove(page, 4); // 玩家選擇格子 4
await makeMove(page, 3); // 玩家選擇格子 3
await makeMove(page, 5); // 玩家選擇格子 5
await makeMove(page, 7); // 玩家選擇格子 7
await makeMove(page, 6); // 玩家選擇格子 6
await makeMove(page, 8); // 玩家選擇格子 8

await expect(page.locator('[data-testid="result-message"]')).toHaveText('這局平手!');
});

// 測試需求 8: 確認遊戲結束後是否正確判斷輸贏是否合法
test('should verify if the game ends with a correct legal win/loss condition', async ({ page }) => {
await page.goto('http://localhost:8080/');
await page.click('[data-testid="choose-X"]');

// 模擬玩家獲勝局面
await makeMove(page, 0); // 玩家選擇格子 0
await makeMove(page, 4); // 玩家選擇格子 4
await makeMove(page, 8); // 玩家選擇格子 8

// 確認遊戲是否正確判斷贏局
const resultMessage = await page.locator('[data-testid="result-message"]').innerText();
expect(resultMessage === '恭喜!你贏了' || resultMessage === '你輸了!下次再努力').toBe(true);
});

// 測試需求 9: 玩家確認結果後遊戲重啟
test('should restart game after confirming result', async ({ page }) => {
await page.goto('http://localhost:8080/');
await page.click('[data-testid="choose-X"]');

await makeMove(page, 0); // 玩家選擇格子 0
await makeMove(page, 4); // 玩家選擇格子 4
await makeMove(page, 8); // 玩家選擇格子 8

await expect(page.locator('[data-testid="result-message"]')).toHaveText('恭喜!你贏了');
await page.click('[data-testid="confirm-result"]');

// 確認遊戲重新開始，並且所有格子清空
const emptyCells = await page.locator('.cell:empty');
expect(emptyCells.count()).toBe(9); // 所有格子應該都為空
});

});
Loading