Concurrency during http requests tests

[WiolaAnd]

I don't know if I am misinterpreting the chart or if my configuration is not working. From the documentation, I understand that maxVusers: x stands for maintaining a specific number of x users during the entire test phase. I interpret them as users competing with each other, that is, executing sequences of requests independently of each other.
The graph generated in the report includes vusers.created, which I understand represents these competing users. If so, their number never exceeds 20VU.

1. Is it my disinterpretation of the graph, or is there really no 50 concurrent competing users in my test? If so, what am I doing wrong in the test setup?

Previously, I was working with JMeter, where 50 concurrent threads were created to execute a sequence of http requests for the duration of the entire test. A thread terminated by an error was replaced by a new thread to maintain the set competition.

2. Is it possible to replicate such a scenario in Artillery? Does a thread have to terminate after performing “flow”?

I tried “loop”, however here it is impossible to indicate that it only lasts for the duration of the test. Either count = x, or never ending, which means it ignores the duration value.

Thanks for your help! I attach the chart and config.

config:
  http:
    extendedMetrics: true
    timeout: 15
  tls:
    rejectUnauthorized: false
  target: ""
  phases:
    - arrivalRate: 10
      duration: 2min
      maxVusers: 50
  payload:
    cast: false
    order: sequence
    path: "./csv/tokens.csv"
    fields:
      - 'token'
  plugins:
    ensure:
      maxErrorRate: 5
    expect:
      reportFailuresAsErrors: true
    metrics-by-endpoint:
      useOnlyRequestNames: true


scenarios:
  - name: "Basic"
    flow:
      - get:
          name: Home
          url: "/"
          expect:
            statusCode: 200
          headers:
            Authorization: "Bearer {{ token }}"
      - get:
          name: Addressbook
          url: "/addressbook"
          expect:
            statusCode: 200
          headers:
            Authorization: "Bearer {{ token }}"
      - get:
          name: Search
          url: "/search"
          expect:
            statusCode: 200
          headers:
            Authorization: "Bearer {{ token }}"
      - get:
          name: NewsList
          url: "/list-news-load-test"
          expect:
            statusCode: 200
          headers:
            Authorization: "Bearer {{ token }}"
      - get:
          name: News
          url: "/list-news-load-test/news-load-test"
          expect:
            statusCode: 200
          headers:
            Authorization: "Bearer {{ token }}"
      - get:
          name: OrganisationStructure
          url: "/our-company/struktura-firmy"
          expect:
            statusCode: 200
          headers:
            Authorization: "Bearer {{ token }}"
      - get:
          name: EventList
          url: "/eventlist-load-test"
          expect:
            statusCode: 200
          headers:
            Authorization: "Bearer {{ token }}"
      - get:
          name: Event
          url: "/eventlist-load-test/event-load-test"
          expect:
            statusCode: 200
          headers:
            Authorization: "Bearer {{ token }}"
      - get:
          name: Page
          url: "/page-load-test"
          expect:
            statusCode: 200
          headers:
            Authorization: "Bearer {{ token }}"
      - get:
          name: Faq
          url: "/faq-load-test"
          expect:
            statusCode: 200
          headers:
            Authorization: "Bearer {{ token }}"

[Ta44ar]

I got common problem 👍

JMeter conf:

how to reproduce test like that in Artillery?

[WiolaAnd]

Looks like we fall into a common pitfall around Artillery newcomers, who are related to any other tools. The thing is - seems like artillery doesnt support closed system model you might know from JMeter, I mean keeping constant amount of users doing ask-wait-catch response sequence.

Looks like @hassy might be tired of clarifying it over and over again so he only formatted the .yml script code here in discussion, but thats understandable for me now lol.

More about workloads modes artillery serves here:
https://www.artillery.io/blog/load-testing-workload-models

[hassy]

@WiolaAnd oooops, sorry - I was in the middle of replying at the time (after fixing YAML formatting) but got pulled away by something else.

Your understanding of maxVusers is correct - it sets the limit on the number of concurrent VUs. The value can be less (it depends on the arrival rate and duration of each VU session), but never more.

But yes, Artillery does not do "iterations" like those in JMeter, where the same VU session gets replayed over and over again by design. Most real-world systems are open/hybrid loop, and using a closed-loop approach to test an open-loop system will lead to performance measurements that are wrong.