Skip to content

Commit

Permalink
Merge pull request #20 from Yuankai619/develop/test1-llm-version
Browse files Browse the repository at this point in the history
Generate test script by LLM
  • Loading branch information
owen0806 authored Sep 19, 2024
2 parents 96b30c7 + 7579b24 commit 15dbee3
Show file tree
Hide file tree
Showing 7 changed files with 821 additions and 11 deletions.
24 changes: 14 additions & 10 deletions test1/playwright.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,17 @@ import { defineConfig, devices } from '@playwright/test';
*/
export default defineConfig({
testDir: './tests',
timeout: 6000,
testMatch:[
'basic.spec.ts',
'llm4.spec.ts'
],
timeout: 8000,
/* Run tests in files in parallel */
fullyParallel: true,
/* Fail the build on CI if you accidentally left test.only in the source code. */
forbidOnly: !!process.env.CI,
/* Retry on CI only */
retries: process.env.CI ? 2 : 0,
retries: process.env.CI ? 2 :2,
/* Opt out of parallel tests on CI. */
workers: process.env.CI ? 1 : undefined,
/* Reporter to use. See https://playwright.dev/docs/test-reporters */
Expand All @@ -39,15 +43,15 @@ export default defineConfig({
use: { ...devices['Desktop Chrome'] },
},

{
name: 'firefox',
use: { ...devices['Desktop Firefox'] },
},
// {
// name: 'firefox',
// use: { ...devices['Desktop Firefox'] },
// },

{
name: 'webkit',
use: { ...devices['Desktop Safari'] },
},
// {
// name: 'webkit',
// use: { ...devices['Desktop Safari'] },
// },

/* Test against mobile viewports. */
// {
Expand Down
45 changes: 44 additions & 1 deletion test1/prompts_record.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,4 +92,47 @@ bug: 一開始談出的選擇視窗樣式錯誤,看不到任何的文字和符
<script src="scripts.js"></script>
</body>
</html>
```
```

---
# 以下是測試prompt

## Prompt1
現在要用playwright進行端對端測試,針對每一個需求進行最詳盡的測試並且給我,預設使用typerscript 去生成playwright腳本,對於每個測試腳本要註解是測試的需求是哪個,並且盡量用data-testid和aria-label去做selector,給我完整的測試腳本

## Prompt2
生成的測試有幾個錯誤
每個動作是隨機的,所以沒辦法透過按照順序按格子模擬玩遊戲,因為有可能已經被電腦選走了
還有每個動作電腦回應都會停頓600ms,所以玩遊戲的過程中都要先等電腦回應後再動作
還有測試上沒有測試到遊戲結束後判斷輸贏是不是合法的

## Prompt3
在測試should randomly decide who goes first有錯誤,因為先手是隨機決定的,因此有可能是後手也有可能是先手,但是判斷卻是空格要小於9,如果是先手的話就會出錯
還有測試should display "Congratulations! You win" when player wins有錯誤,因為電腦是隨機選格子,所以如果直接模擬寫死的步驟碰巧被電腦選走就會出錯
should display "You lose! Try again" when player loses也是一樣,沒辦法透過簡單寫死的步驟讓電腦贏
should display "It\'s a tie!" when there is a tie也是,寫死的步驟會難以保證結果是平手的
should verify if the game ends with a correct legal win/loss condition也是沒辦法透過寫死的步驟保證這個是贏局或輸局,也有可能平手或是還沒結束
should restart game after confirming result 也是一樣的問題,寫死的步驟不能確保遊戲結束

## Prompt4
在should randomly decide who goes first會有錯誤: Error: expect(received).toBeGreaterThanOrEqual(expected)

Matcher error: received value must be a number or bigint

Received has type: object
Received has value: {Symbol(async_id_symbol): 5177, Symbol(trigger_async_id_symbol): 5176, Symbol(kResourceStore): undefined}
還有should not allow painting over existing moves會有錯誤:
Error: expect(received).toBeLessThan(expected)

Matcher error: received value must be a number or bigint

Received has type: object
Received has value: {Symbol(async_id_symbol): 7091, Symbol(trigger_async_id_symbol): 7090, Symbol(kResourceStore): undefined}
還有should restart game after confirming result有錯誤:
Error: expect(received).toBe(expected) // Object.is equality

Expected: 9
Received: {Symbol(async_id_symbol): 20900, Symbol(trigger_async_id_symbol): 20899, Symbol(kResourceStore): undefined}
還有在測試最後結果要輸、贏、平手的測試遊玩過程雖然正確但是電腦是隨機的,所以真的輸贏情形是無法預測的,所以應該要改成由玩遊戲然後判斷遊戲結果和board上面的紀錄是不是符合輸贏的條件才對
## Prompt5
還是有一些錯,在測試should display "It\'s a tie!" when there is a tie、should display "You lose! Try again" when player loses、其實是在測試同個動作,所以可以結合成一個就好
146 changes: 146 additions & 0 deletions test1/tests/llm0.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
import { test, expect } from '@playwright/test';

test.describe('Tic-Tac-Toe Game', () => {

// 測試需求 1: 確認遊戲開始時會顯示選擇符號的彈跳視窗
test('should display symbol choice modal at the beginning', async ({ page }) => {
// Navigate to the game
await page.goto('http://localhost:8080/');

// 確認彈出視窗顯示,且標題正確
const modalTitle = await page.locator('[data-testid="modal-title"]');
await expect(modalTitle).toHaveText('選擇你的符號');

// 確認按鈕 X 和 O 存在
await expect(page.locator('[data-testid="choose-X"]')).toBeVisible();
await expect(page.locator('[data-testid="choose-O"]')).toBeVisible();
});

// 測試需求 2: 玩家選擇 X 或 O 並開始遊戲
test('should start game after symbol selection', async ({ page }) => {
await page.goto('http://localhost:8080/');

// 點擊選擇 X 按鈕
await page.click('[data-testid="choose-X"]');

// 確認棋盤可見,並且沒有選擇彈出視窗
await expect(page.locator('[data-testid="game-board"]')).toBeVisible();
await expect(page.locator('[data-testid="symbol-choice-modal"]')).toBeHidden();
});

// 測試需求 3: 隨機決定玩家或電腦誰先畫格子
test('should randomly decide who goes first', async ({ page }) => {
await page.goto('http://localhost:8080/');

// 選擇 X 開始遊戲
await page.click('[data-testid="choose-X"]');

// 確認遊戲啟動後有至少一個棋格被選中(電腦可能會先行動)
const emptyCells = await page.locator('.cell:empty');
expect(emptyCells.count()).toBeLessThan(9); // 確認有格子已被畫上
});

// 測試需求 4: 玩家與電腦交互畫格子,不允許覆蓋已畫的格子
test('should not allow painting over existing moves', async ({ page }) => {
await page.goto('http://localhost:8080/');

// 玩家選擇 X 開始遊戲
await page.click('[data-testid="choose-X"]');

// 玩家選擇格子 0
const cell0 = page.locator('[data-testid="cell-0"]');
await cell0.click();

// 試圖再次點擊同一格子
await cell0.click();

// 確認內容未改變
await expect(cell0).toHaveText('X');
});

// 測試需求 5: 玩家獲勝時彈出視窗顯示 "恭喜!你贏了"
test('should display "Congratulations! You win" when player wins', async ({ page }) => {
await page.goto('http://localhost:8080/');

// 選擇 X 開始遊戲
await page.click('[data-testid="choose-X"]');

// 模擬玩家贏局,依次選擇 0, 1, 2
await page.click('[data-testid="cell-0"]'); // Player
await page.click('[data-testid="cell-3"]'); // Computer (random)
await page.click('[data-testid="cell-1"]'); // Player
await page.click('[data-testid="cell-4"]'); // Computer (random)
await page.click('[data-testid="cell-2"]'); // Player wins

// 確認贏的訊息彈出
await expect(page.locator('[data-testid="result-message"]')).toHaveText('恭喜!你贏了');
});

// 測試需求 6: 玩家失敗時彈出視窗顯示 "你輸了!下次再努力"
test('should display "You lose! Try again" when player loses', async ({ page }) => {
await page.goto('http://localhost:8080/');

// 選擇 X 開始遊戲
await page.click('[data-testid="choose-X"]');

// 模擬電腦贏局,依次選擇使電腦獲勝的步驟
await page.click('[data-testid="cell-0"]'); // Player
await page.click('[data-testid="cell-3"]'); // Computer
await page.click('[data-testid="cell-1"]'); // Player
await page.click('[data-testid="cell-4"]'); // Computer
await page.click('[data-testid="cell-8"]'); // Player
await page.click('[data-testid="cell-5"]'); // Computer wins

// 確認輸的訊息彈出
await expect(page.locator('[data-testid="result-message"]')).toHaveText('你輸了!下次再努力');
});

// 測試需求 7: 平局時顯示 "這局平手!"
test('should display "It\'s a tie!" when there is a tie', async ({ page }) => {
await page.goto('http://localhost:8080/');

// 選擇 X 開始遊戲
await page.click('[data-testid="choose-X"]');

// 模擬平局
await page.click('[data-testid="cell-0"]'); // Player
await page.click('[data-testid="cell-1"]'); // Computer
await page.click('[data-testid="cell-2"]'); // Player
await page.click('[data-testid="cell-4"]'); // Computer
await page.click('[data-testid="cell-3"]'); // Player
await page.click('[data-testid="cell-5"]'); // Computer
await page.click('[data-testid="cell-7"]'); // Player
await page.click('[data-testid="cell-6"]'); // Computer
await page.click('[data-testid="cell-8"]'); // Player

// 確認平局訊息彈出
await expect(page.locator('[data-testid="result-message"]')).toHaveText('這局平手!');
});

// 測試需求 8: 玩家確認結果後遊戲重啟
test('should restart game after confirming result', async ({ page }) => {
await page.goto('http://localhost:8080/');

// 選擇 X 開始遊戲
await page.click('[data-testid="choose-X"]');

// 模擬玩家贏局
await page.click('[data-testid="cell-0"]'); // Player
await page.click('[data-testid="cell-3"]'); // Computer (random)
await page.click('[data-testid="cell-1"]'); // Player
await page.click('[data-testid="cell-4"]'); // Computer (random)
await page.click('[data-testid="cell-2"]'); // Player wins

// 確認贏的訊息彈出
await expect(page.locator('[data-testid="result-message"]')).toHaveText('恭喜!你贏了');

// 點擊確認按鈕重新開始遊戲
await page.click('[data-testid="confirm-result"]');

// 確認遊戲已重新開始
await expect(page.locator('[data-testid="game-board"]')).toBeVisible();
const emptyCells = await page.locator('.cell:empty');
expect(emptyCells.count()).toBe(9); // 所有格子應該都是空的
});

});
134 changes: 134 additions & 0 deletions test1/tests/llm1.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
import { test, expect } from '@playwright/test';

// Helper function to simulate player moves
async function makeMove(page, cellIndex) {
const cell = page.locator(`[data-testid="cell-${cellIndex}"]`);
const isCellEmpty = await cell.evaluate(node => node.textContent === '');

if (isCellEmpty) {
await cell.click();
await page.waitForTimeout(600); // 等待電腦做出回應
}
}

test.describe('Tic-Tac-Toe Game', () => {

// 測試需求 1: 確認遊戲開始時會顯示選擇符號的彈跳視窗
test('should display symbol choice modal at the beginning', async ({ page }) => {
await page.goto('http://localhost:8080/');

const modalTitle = await page.locator('[data-testid="modal-title"]');
await expect(modalTitle).toHaveText('選擇你的符號');
await expect(page.locator('[data-testid="choose-X"]')).toBeVisible();
await expect(page.locator('[data-testid="choose-O"]')).toBeVisible();
});

// 測試需求 2: 玩家選擇 X 或 O 並開始遊戲
test('should start game after symbol selection', async ({ page }) => {
await page.goto('http://localhost:8080/');
await page.click('[data-testid="choose-X"]');

await expect(page.locator('[data-testid="game-board"]')).toBeVisible();
await expect(page.locator('[data-testid="symbol-choice-modal"]')).toBeHidden();
});

// 測試需求 3: 隨機決定玩家或電腦誰先畫格子
test('should randomly decide who goes first', async ({ page }) => {
await page.goto('http://localhost:8080/');
await page.click('[data-testid="choose-X"]');

await page.waitForTimeout(600); // 等待電腦可能的回應
const emptyCells = await page.locator('.cell:empty');
expect(emptyCells.count()).toBeLessThan(9); // 確認有格子已被畫上
});

// 測試需求 4: 玩家與電腦交互畫格子,不允許覆蓋已畫的格子
test('should not allow painting over existing moves', async ({ page }) => {
await page.goto('http://localhost:8080/');
await page.click('[data-testid="choose-X"]');

await makeMove(page, 0); // 玩家選擇格子 0

const cell0 = page.locator('[data-testid="cell-0"]');
await cell0.click(); // 試圖再次點擊已選的格子
await expect(cell0).toHaveText('X');
});

// 測試需求 5: 玩家獲勝時顯示 "恭喜!你贏了"
test('should display "Congratulations! You win" when player wins', async ({ page }) => {
await page.goto('http://localhost:8080/');
await page.click('[data-testid="choose-X"]');

await makeMove(page, 0); // 玩家選擇格子 0
await makeMove(page, 4); // 玩家選擇格子 4
await makeMove(page, 8); // 玩家選擇格子 8

// 檢查是否出現勝利訊息
await expect(page.locator('[data-testid="result-message"]')).toHaveText('恭喜!你贏了');
});

// 測試需求 6: 玩家失敗時顯示 "你輸了!下次再努力"
test('should display "You lose! Try again" when player loses', async ({ page }) => {
await page.goto('http://localhost:8080/');
await page.click('[data-testid="choose-X"]');

// 玩家和電腦隨機下棋,這裡模擬輸局
await makeMove(page, 0); // 玩家選擇格子 0
await makeMove(page, 2); // 玩家選擇格子 2
await makeMove(page, 8); // 玩家選擇格子 8

// 確認失敗訊息
await expect(page.locator('[data-testid="result-message"]')).toHaveText('你輸了!下次再努力');
});

// 測試需求 7: 平局時顯示 "這局平手!"
test('should display "It\'s a tie!" when there is a tie', async ({ page }) => {
await page.goto('http://localhost:8080/');
await page.click('[data-testid="choose-X"]');

await makeMove(page, 0); // 玩家選擇格子 0
await makeMove(page, 1); // 玩家選擇格子 1
await makeMove(page, 2); // 玩家選擇格子 2
await makeMove(page, 4); // 玩家選擇格子 4
await makeMove(page, 3); // 玩家選擇格子 3
await makeMove(page, 5); // 玩家選擇格子 5
await makeMove(page, 7); // 玩家選擇格子 7
await makeMove(page, 6); // 玩家選擇格子 6
await makeMove(page, 8); // 玩家選擇格子 8

await expect(page.locator('[data-testid="result-message"]')).toHaveText('這局平手!');
});

// 測試需求 8: 確認遊戲結束後是否正確判斷輸贏是否合法
test('should verify if the game ends with a correct legal win/loss condition', async ({ page }) => {
await page.goto('http://localhost:8080/');
await page.click('[data-testid="choose-X"]');

// 模擬玩家獲勝局面
await makeMove(page, 0); // 玩家選擇格子 0
await makeMove(page, 4); // 玩家選擇格子 4
await makeMove(page, 8); // 玩家選擇格子 8

// 確認遊戲是否正確判斷贏局
const resultMessage = await page.locator('[data-testid="result-message"]').innerText();
expect(resultMessage === '恭喜!你贏了' || resultMessage === '你輸了!下次再努力').toBe(true);
});

// 測試需求 9: 玩家確認結果後遊戲重啟
test('should restart game after confirming result', async ({ page }) => {
await page.goto('http://localhost:8080/');
await page.click('[data-testid="choose-X"]');

await makeMove(page, 0); // 玩家選擇格子 0
await makeMove(page, 4); // 玩家選擇格子 4
await makeMove(page, 8); // 玩家選擇格子 8

await expect(page.locator('[data-testid="result-message"]')).toHaveText('恭喜!你贏了');
await page.click('[data-testid="confirm-result"]');

// 確認遊戲重新開始,並且所有格子清空
const emptyCells = await page.locator('.cell:empty');
expect(emptyCells.count()).toBe(9); // 所有格子應該都為空
});

});
Loading

0 comments on commit 15dbee3

Please sign in to comment.