Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate test script by LLM #22

Merged
merged 1 commit into from
Sep 19, 2024
Merged

Generate test script by LLM #22

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Generate test script by LLM
  • Loading branch information
@zihan0221
zihan0221 committed Sep 19, 2024
commit 4343a060c350de07d3c4c81403590551e2e5acdd
2 changes: 1 addition & 1 deletion test1/playwright.config.ts
View file
Original file line number Diff line number Diff line change
@@ -22,7 +22,7 @@ export default defineConfig({
/* Fail the build on CI if you accidentally left test.only in the source code. */
forbidOnly: !!process.env.CI,
/* Retry on CI only */
retries: process.env.CI ? 2 :2,
retries: process.env.CI ? 2 :0,
/* Opt out of parallel tests on CI. */
workers: process.env.CI ? 1 : undefined,
/* Reporter to use. See https://playwright.dev/docs/test-reporters */
Binary file added test3/images/image-4.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 6 additions & 2 deletions test3/playwright.config.ts
View file
Original file line number Diff line number Diff line change
@@ -12,6 +12,10 @@ import { defineConfig, devices } from '@playwright/test';
*/
export default defineConfig({
testDir: './tests',
testMatch:[
'basic.spec.ts',
'llm4.spec.ts'
],
/* Run tests in files in parallel */
fullyParallel: true,
/* Fail the build on CI if you accidentally left test.only in the source code. */
@@ -39,15 +43,15 @@ export default defineConfig({
use: { ...devices['Desktop Chrome'] },
},

{
/*{
name: 'firefox',
use: { ...devices['Desktop Firefox'] },
},

{
name: 'webkit',
use: { ...devices['Desktop Safari'] },
},
},*/

/* Test against mobile viewports. */
// {
35 changes: 33 additions & 2 deletions test3/prompts_record.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
## P0 Prompt 1
## P0 Prompt 0
我直接把「飲料訂購系統.pdf」塞給claude
### 成果
![alt text](./images/image.png)
@@ -10,7 +10,7 @@ bug: 飲料訂購那些資訊需要塞成一個橫排的表格，如pdf中圖3-1
不算bug的bug: 題目會給一個csv，但我沒有丟csv給claude，所以他在js那邊模擬了一個csv，很有想法


## P0 Prompt 2
## P0 Prompt 1
按照成果1的bug去修，我丟給他以下截圖給的Prompt為「我希望飲料訂購部分的排版跟這個一樣，並重新給我完整的html5,css,js檔案」
![alt text](./images/image-2.png)

@@ -25,3 +25,34 @@ bug: 飲料訂購那些資訊需要塞成一個橫排的表格，如pdf中圖3-1
### 成果
跟上一個成果沒什麼不一樣

### p2 Prompt 0
我把「html5,css,js」丟給Claude，並且給他這樣的Prompt:「現在要用playwright進行端對端測試，針對每一個需求進行最詳盡的測試並且給我，預設使用typerscript 去生成playwright腳本，對於每個測試腳本要註解是測試的需求是哪個，並且盡量用data-testid和aria-label去做selector，給我完整的測試腳本」

### 成果
Claude給了這些測試腳本，有些部分測試會是爛掉的，像是「溫度選項正確」、「甜度選項正確」、「結帳流程」，因此我讓Claude一個一個慢慢改

![alt text](./images/image-4.png)

### p2 Prompt 1 (修改「溫度選項正確」)
Prompt:「溫度選項正確這一個測試是錯的，我要好好的打開combobox，去看裡面的東西是不是為visible的，請幫我修改這個測試」

### 成果
「溫度選項正確」是對的

### p2 Prompt 2(修改「甜度選項正確」)
Prompt:「甜度選項正確這一個測試是錯的，我要好好的打開combobox，去看裡面的東西是不是為visible的，請幫我修改這個測試」

### 成果
「甜度選項正確」是錯的，他根本沒有好好的打開combobox接著check裡面的東西

### p2 Prompt 3(修改「甜度選項正確」)
在check visible的時候出現問題，請麻煩改正它

### 成果
「溫度選項正確」是對的

### p2 Prompt 4修改「結帳流程顯示訂單摘要並清空訂購清單」
最後一個測試：「結帳流程顯示訂單摘要並清空訂購清單」也是錯的，在checkoutButton.click時出了問題，請修他

### 成果
「結帳流程顯示訂單摘要並清空訂購清單」對了，可喜可賀
140 changes: 140 additions & 0 deletions test3/tests/llm0.spec.ts
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
import { test, expect } from '@playwright/test';

test.describe('飲料訂購系統', () => {
test.beforeEach(async ({ page }) => {
await page.goto('http://localhost:8080'); // 根據需要調整 URL
});

// 需求 1: 使用者界面元素
test('使用者界面元素存在且正確佈局', async ({ page }) => {
// 檢查主要元素是否存在
await expect(page.getByRole('heading', { name: '來點什麼...' })).toBeVisible();
await expect(page.getByTestId('price-display')).toBeVisible();
await expect(page.getByTestId('total-display')).toBeVisible();
await expect(page.getByRole('button', { name: '確認' })).toBeVisible();

// 檢查下拉式選單元素
await expect(page.getByRole('combobox', { name: '品名' })).toBeVisible();
await expect(page.getByRole('combobox', { name: '容量' })).toBeVisible();
await expect(page.getByRole('combobox', { name: '溫度' })).toBeVisible();
await expect(page.getByRole('combobox', { name: '甜度' })).toBeVisible();

// 檢查輸入元素
await expect(page.getByRole('spinbutton', { name: '數量' })).toBeVisible();
await expect(page.getByRole('textbox', { name: '訂購人' })).toBeVisible();
});

// 需求 2a, 2b, 2c: 品名和容量選項
test('品名和容量選項正確填充', async ({ page }) => {
const itemSelect = page.getByRole('combobox', { name: '品名' });
await expect(itemSelect).toHaveCount(1);

const itemOptions = await itemSelect.getByRole('option').all();
expect(itemOptions.length).toBeGreaterThan(0);
expect(itemOptions.length).toBeLessThanOrEqual(10);

// 選擇第一個品項並檢查容量選項
await itemSelect.selectOption({ index: 0 });
const sizeSelect = page.getByRole('combobox', { name: '容量' });
const sizeOptions = await sizeSelect.getByRole('option').all();
expect(sizeOptions.length).toBeGreaterThan(0);
});

// 需求 2d, 3: 價格和總金額計算
test('價格和總金額計算正確', async ({ page }) => {
const itemSelect = page.getByRole('combobox', { name: '品名' });
const sizeSelect = page.getByRole('combobox', { name: '容量' });
const quantityInput = page.getByRole('spinbutton', { name: '數量' });
const priceDisplay = page.getByTestId('price-display');
const totalDisplay = page.getByTestId('total-display');

await itemSelect.selectOption({ index: 0 });
await sizeSelect.selectOption({ index: 0 });
await expect(priceDisplay).not.toHaveText('0');

await quantityInput.fill('2');
const price = await priceDisplay.innerText();
const expectedTotal = (parseInt(price) * 2).toString();
await expect(totalDisplay).toHaveText(expectedTotal);
});

// 需求 4: 溫度選項
test('溫度選項正確', async ({ page }) => {
const tempSelect = page.getByRole('combobox', { name: '溫度' });
const options = ['熱', '全冰', '少冰', '去冰'];
for (const option of options) {
await expect(tempSelect.getByText(option)).toBeVisible();
}
});

// 需求 5: 甜度選項
test('甜度選項正確', async ({ page }) => {
const sweetSelect = page.getByRole('combobox', { name: '甜度' });
const options = ['全糖', '8分糖', '5分糖', '3分糖', '1分糖', '無糖'];
for (const option of options) {
await expect(sweetSelect.getByText(option)).toBeVisible();
}
});

// 需求 6, 7: 訂單確認和顯示在訂購清單中
test('確認後訂單添加到訂購清單', async ({ page }) => {
await fillOrderForm(page);
await page.getByRole('button', { name: '確認' }).click();
const orderList = page.getByTestId('order-list');
await expect(orderList.getByRole('row')).toHaveCount(2); // 表頭 + 1 個訂單
await expect(orderList.getByRole('checkbox')).toBeVisible();
});

// 需求 8: 刪除選定的訂單
test('可以刪除選定的訂單', async ({ page }) => {
await fillOrderForm(page);
await page.getByRole('button', { name: '確認' }).click();
await page.getByTestId('order-list').getByRole('checkbox').check();
await page.getByRole('button', { name: '刪除勾選' }).click();
await expect(page.getByTestId('order-list').getByRole('row')).toHaveCount(1); // 只剩表頭
});

// 需求 9: 訂單摘要
test('訂單摘要正確更新', async ({ page }) => {
await fillOrderForm(page);
await page.getByRole('button', { name: '確認' }).click();
const summary = page.getByTestId('order-summary');
await expect(summary).toContainText('總數量: 1');
await expect(summary).toContainText('總金額: 30');
});

// 需求 10: 自動填充客戶姓名
test('第一筆訂單後自動填充客戶姓名', async ({ page }) => {
await fillOrderForm(page, '測試客戶');
await page.getByRole('button', { name: '確認' }).click();
await fillOrderForm(page);
const customerInput = page.getByRole('textbox', { name: '訂購人' });
await expect(customerInput).toHaveValue('測試客戶');
});

// 需求 11: 結帳流程
test('結帳流程顯示訂單摘要並清空訂購清單', async ({ page }) => {
await fillOrderForm(page);
await page.getByRole('button', { name: '確認' }).click();
const checkoutButton = page.getByRole('button', { name: '結帳' });

const dialogPromise = page.waitForEvent('dialog');
await checkoutButton.click();
const dialog = await dialogPromise;

expect(dialog.message()).toContain('訂單摘要');
expect(dialog.message()).toContain('總金額');

await dialog.dismiss();
await expect(page.getByTestId('order-list').getByRole('row')).toHaveCount(1); // 只剩表頭
});
});

async function fillOrderForm(page, customerName = '測試客戶') {
await page.getByRole('combobox', { name: '品名' }).selectOption({ index: 0 });
await page.getByRole('combobox', { name: '容量' }).selectOption({ index: 0 });
await page.getByRole('spinbutton', { name: '數量' }).fill('1');
await page.getByRole('combobox', { name: '溫度' }).selectOption('全冰');
await page.getByRole('combobox', { name: '甜度' }).selectOption('全糖');
await page.getByRole('textbox', { name: '訂購人' }).fill(customerName);
}
155 changes: 155 additions & 0 deletions test3/tests/llm1.spec.ts
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
import { test, expect } from '@playwright/test';

test.describe('飲料訂購系統', () => {
test.beforeEach(async ({ page }) => {
await page.goto('http://localhost:8080'); // 根據需要調整 URL
});

// 需求 1: 使用者界面元素
test('使用者界面元素存在且正確佈局', async ({ page }) => {
// 檢查主要元素是否存在
await expect(page.getByRole('heading', { name: '來點什麼...' })).toBeVisible();
await expect(page.getByTestId('price-display')).toBeVisible();
await expect(page.getByTestId('total-display')).toBeVisible();
await expect(page.getByRole('button', { name: '確認' })).toBeVisible();

// 檢查下拉式選單元素
await expect(page.getByRole('combobox', { name: '品名' })).toBeVisible();
await expect(page.getByRole('combobox', { name: '容量' })).toBeVisible();
await expect(page.getByRole('combobox', { name: '溫度' })).toBeVisible();
await expect(page.getByRole('combobox', { name: '甜度' })).toBeVisible();

// 檢查輸入元素
await expect(page.getByRole('spinbutton', { name: '數量' })).toBeVisible();
await expect(page.getByRole('textbox', { name: '訂購人' })).toBeVisible();
});

// 需求 2a, 2b, 2c: 品名和容量選項
test('品名和容量選項正確填充', async ({ page }) => {
const itemSelect = page.getByRole('combobox', { name: '品名' });
await expect(itemSelect).toHaveCount(1);

const itemOptions = await itemSelect.getByRole('option').all();
expect(itemOptions.length).toBeGreaterThan(0);
expect(itemOptions.length).toBeLessThanOrEqual(10);

// 選擇第一個品項並檢查容量選項
await itemSelect.selectOption({ index: 0 });
const sizeSelect = page.getByRole('combobox', { name: '容量' });
const sizeOptions = await sizeSelect.getByRole('option').all();
expect(sizeOptions.length).toBeGreaterThan(0);
});

// 需求 2d, 3: 價格和總金額計算
test('價格和總金額計算正確', async ({ page }) => {
const itemSelect = page.getByRole('combobox', { name: '品名' });
const sizeSelect = page.getByRole('combobox', { name: '容量' });
const quantityInput = page.getByRole('spinbutton', { name: '數量' });
const priceDisplay = page.getByTestId('price-display');
const totalDisplay = page.getByTestId('total-display');

await itemSelect.selectOption({ index: 0 });
await sizeSelect.selectOption({ index: 0 });
await expect(priceDisplay).not.toHaveText('0');

await quantityInput.fill('2');
const price = await priceDisplay.innerText();
const expectedTotal = (parseInt(price) * 2).toString();
await expect(totalDisplay).toHaveText(expectedTotal);
});

// 需求 4: 溫度選項
test('溫度選項正確', async ({ page }) => {
const tempSelect = page.getByRole('combobox', { name: '溫度' });

// 等待選擇器元素出現並可見
await expect(tempSelect).toBeVisible({ timeout: 5000 });

const expectedOptions = ['熱', '全冰', '少冰', '去冰'];

// 獲取實際的選項
const actualOptions = await tempSelect.getByRole('option').allInnerTexts();

// 檢查所有預期的選項是否都存在，忽略順序
for (const option of expectedOptions) {
expect(actualOptions).toContain(option);
}

// 檢查選項數量是否正確
expect(actualOptions.length).toBe(expectedOptions.length);

// 如果需要檢查順序，可以使用以下代碼：
// expect(actualOptions).toEqual(expectedOptions);
});

// 需求 5: 甜度選項
test('甜度選項正確', async ({ page }) => {
const sweetSelect = page.getByRole('combobox', { name: '甜度' });
const options = ['全糖', '8分糖', '5分糖', '3分糖', '1分糖', '無糖'];
for (const option of options) {
await expect(sweetSelect.getByText(option)).toBeVisible();
}
});

// 需求 6, 7: 訂單確認和顯示在訂購清單中
test('確認後訂單添加到訂購清單', async ({ page }) => {
await fillOrderForm(page);
await page.getByRole('button', { name: '確認' }).click();
const orderList = page.getByTestId('order-list');
await expect(orderList.getByRole('row')).toHaveCount(2); // 表頭 + 1 個訂單
await expect(orderList.getByRole('checkbox')).toBeVisible();
});

// 需求 8: 刪除選定的訂單
test('可以刪除選定的訂單', async ({ page }) => {
await fillOrderForm(page);
await page.getByRole('button', { name: '確認' }).click();
await page.getByTestId('order-list').getByRole('checkbox').check();
await page.getByRole('button', { name: '刪除勾選' }).click();
await expect(page.getByTestId('order-list').getByRole('row')).toHaveCount(1); // 只剩表頭
});

// 需求 9: 訂單摘要
test('訂單摘要正確更新', async ({ page }) => {
await fillOrderForm(page);
await page.getByRole('button', { name: '確認' }).click();
const summary = page.getByTestId('order-summary');
await expect(summary).toContainText('總數量: 1');
await expect(summary).toContainText('總金額: 30');
});

// 需求 10: 自動填充客戶姓名
test('第一筆訂單後自動填充客戶姓名', async ({ page }) => {
await fillOrderForm(page, '測試客戶');
await page.getByRole('button', { name: '確認' }).click();
await fillOrderForm(page);
const customerInput = page.getByRole('textbox', { name: '訂購人' });
await expect(customerInput).toHaveValue('測試客戶');
});

// 需求 11: 結帳流程
test('結帳流程顯示訂單摘要並清空訂購清單', async ({ page }) => {
await fillOrderForm(page);
await page.getByRole('button', { name: '確認' }).click();
const checkoutButton = page.getByRole('button', { name: '結帳' });

const dialogPromise = page.waitForEvent('dialog');
await checkoutButton.click();
const dialog = await dialogPromise;

expect(dialog.message()).toContain('訂單摘要');
expect(dialog.message()).toContain('總金額');

await dialog.dismiss();
await expect(page.getByTestId('order-list').getByRole('row')).toHaveCount(1); // 只剩表頭
});
});

async function fillOrderForm(page, customerName = '測試客戶') {
await page.getByRole('combobox', { name: '品名' }).selectOption({ index: 0 });
await page.getByRole('combobox', { name: '容量' }).selectOption({ index: 0 });
await page.getByRole('spinbutton', { name: '數量' }).fill('1');
await page.getByRole('combobox', { name: '溫度' }).selectOption('全冰');
await page.getByRole('combobox', { name: '甜度' }).selectOption('全糖');
await page.getByRole('textbox', { name: '訂購人' }).fill(customerName);
}
Loading